Nov 03 00:02:28.918202 ip-10-0-140-172 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 00:02:28.918218 ip-10-0-140-172 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 00:02:28.918228 ip-10-0-140-172 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 00:02:28.918542 ip-10-0-140-172 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 00:02:40.202462 ip-10-0-140-172 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 00:02:40.202482 ip-10-0-140-172 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 97d4bf86a87842b9984d23e13667ea98 -- Nov 03 00:04:37.440060 ip-10-0-140-172 systemd[1]: Starting Kubernetes Kubelet... Nov 03 00:04:37.849997 ip-10-0-140-172 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:04:37.849997 ip-10-0-140-172 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 00:04:37.849997 ip-10-0-140-172 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:04:37.849997 ip-10-0-140-172 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:04:37.856369 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856253 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:37.856369 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856369 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856373 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856376 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856379 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856382 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856384 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856386 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856389 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856392 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856395 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856397 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856400 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856402 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856409 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856412 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856415 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856417 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856420 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856422 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856425 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:37.856446 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856427 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856430 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856432 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856434 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856437 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856439 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856441 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856444 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856446 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856449 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856451 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856454 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856457 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856459 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856462 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856464 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856467 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856470 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856475 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:37.856915 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856477 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856480 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856483 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856485 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856487 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856490 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856493 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856495 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856498 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856500 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856502 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856505 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856507 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856509 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856512 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856515 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856518 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856520 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856523 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856525 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:37.857339 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856528 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856530 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856533 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856536 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856539 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856541 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856544 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856546 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856549 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856551 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856553 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856556 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856558 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856560 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856562 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856565 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856567 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856570 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856572 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856575 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:37.857814 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856578 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856580 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856584 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856587 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856590 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856592 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856595 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856599 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856602 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856604 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856607 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856609 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856611 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856614 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856616 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856618 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856621 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856624 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856627 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:37.858268 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856629 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856631 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856634 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856636 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856638 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856641 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.856643 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857387 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857394 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857397 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857399 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857402 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857404 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857407 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857410 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857412 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857414 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857417 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857419 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857422 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:37.858697 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857425 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857428 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857430 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857433 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857435 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857437 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857440 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857443 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857445 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857448 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857450 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857453 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857455 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857458 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857460 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857462 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857465 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857467 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857469 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:37.859248 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857471 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857474 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857476 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857478 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857481 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857483 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857486 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857488 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857491 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857493 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857495 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857498 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857501 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857503 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857506 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857508 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857511 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857514 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857516 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:37.859698 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857518 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857521 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857523 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857525 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857528 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857530 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857533 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857535 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857537 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857540 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857542 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857545 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857547 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857549 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857553 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857556 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857559 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857562 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857564 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857567 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:37.860153 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857569 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857572 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857576 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857580 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857583 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857586 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857590 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857592 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857595 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857598 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857600 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857603 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857605 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857608 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857611 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857613 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857615 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857617 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857620 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:37.860642 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857623 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857625 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857627 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857630 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857632 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857634 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857637 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857639 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857642 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857644 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857646 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857649 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857652 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857654 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857656 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.857660 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858149 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858155 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858159 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858163 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:37.861101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858165 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858168 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858171 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858173 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858176 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858178 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858182 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858186 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858189 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858192 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858195 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858197 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858200 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858202 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858204 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858207 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858209 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858212 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858214 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:37.861550 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858217 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858219 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858221 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858224 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858226 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858228 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858231 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858233 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858236 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858239 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858241 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858244 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858247 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858249 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858252 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858254 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858257 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858260 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858262 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858264 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:37.862003 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858267 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858270 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858272 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858275 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858277 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858280 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858282 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858284 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858287 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858289 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858291 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858294 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858296 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858298 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858301 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858303 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858305 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858309 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858311 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858314 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:37.862452 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858316 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858319 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858321 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858324 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858326 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858329 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858331 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858333 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858336 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858338 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858341 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858343 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858345 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858348 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858350 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858354 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858357 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858360 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858363 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:37.862912 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858365 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858368 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858370 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858372 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858374 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858377 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858379 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858381 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858383 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858386 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858388 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858391 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858393 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858395 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858399 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858401 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858404 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858409 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858411 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:37.863375 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858413 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858416 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858418 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858421 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.858423 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858511 2569 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858517 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858523 2569 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858528 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858533 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858536 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858539 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858543 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858546 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858549 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858552 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858555 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858558 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858561 2569 flags.go:64] FLAG: --cgroup-root="" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858564 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858567 2569 flags.go:64] FLAG: --client-ca-file="" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858570 2569 flags.go:64] FLAG: --cloud-provider="external" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858573 2569 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858577 2569 flags.go:64] FLAG: --cluster-domain="" Nov 03 00:04:37.863873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858580 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858583 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858586 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858589 2569 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858592 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858596 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858600 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858604 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858606 2569 flags.go:64] FLAG: --contention-profiling="false" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858609 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858612 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858615 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858618 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858622 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858626 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858629 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858631 2569 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858634 2569 flags.go:64] FLAG: --enable-server="true" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858637 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858641 2569 flags.go:64] FLAG: --event-burst="100" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858644 2569 flags.go:64] FLAG: --event-qps="50" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858647 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858650 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858652 2569 flags.go:64] FLAG: --eviction-hard="" Nov 03 00:04:37.864441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858656 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858659 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858662 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858665 2569 flags.go:64] FLAG: --eviction-soft="" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858667 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858671 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858674 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858677 2569 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858679 2569 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858682 2569 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858685 2569 flags.go:64] FLAG: --feature-gates="" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858689 2569 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858692 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858695 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858700 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858703 2569 flags.go:64] FLAG: --healthz-port="10248" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858706 2569 flags.go:64] FLAG: --help="false" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858709 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858712 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858715 2569 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858718 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858721 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858726 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 00:04:37.865078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858729 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858731 2569 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858734 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858737 2569 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858740 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858742 2569 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858745 2569 flags.go:64] FLAG: --kube-reserved="" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858748 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858750 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858753 2569 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858756 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858759 2569 flags.go:64] FLAG: --lock-file="" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858762 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858765 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858767 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858786 2569 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858789 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858792 2569 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858794 2569 flags.go:64] FLAG: --logging-format="text" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858797 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858800 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858803 2569 flags.go:64] FLAG: --manifest-url="" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858806 2569 flags.go:64] FLAG: --manifest-url-header="" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858811 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858815 2569 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 00:04:37.865615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858819 2569 flags.go:64] FLAG: --max-pods="110" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858822 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858825 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858828 2569 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858830 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858833 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858837 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858840 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858846 2569 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858849 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858852 2569 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858855 2569 flags.go:64] FLAG: --pod-cidr="" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858857 2569 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858860 2569 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858863 2569 flags.go:64] FLAG: --pods-per-core="0" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858866 2569 flags.go:64] FLAG: --port="10250" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858868 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858871 2569 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a9fcc8855f4d508a" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858874 2569 flags.go:64] FLAG: --qos-reserved="" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858877 2569 flags.go:64] FLAG: --read-only-port="10255" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858880 2569 flags.go:64] FLAG: --register-node="true" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858882 2569 flags.go:64] FLAG: --register-with-taints="" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858886 2569 flags.go:64] FLAG: --registry-burst="10" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858889 2569 flags.go:64] FLAG: --registry-qps="5" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858891 2569 flags.go:64] FLAG: --reserved-cpus="" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858894 2569 flags.go:64] FLAG: --reserved-memory="" Nov 03 00:04:37.866207 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858897 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858900 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858903 2569 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858906 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858912 2569 flags.go:64] FLAG: --runonce="false" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858915 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858918 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858922 2569 flags.go:64] FLAG: --seccomp-default="false" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858924 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858927 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858930 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858933 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858937 2569 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858940 2569 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858942 2569 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858945 2569 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858948 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858951 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858954 2569 flags.go:64] FLAG: --system-cgroups="" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858956 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858961 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858964 2569 flags.go:64] FLAG: --tls-cert-file="" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858967 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858971 2569 flags.go:64] FLAG: --tls-min-version="" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858974 2569 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 00:04:37.866821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858977 2569 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858980 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858982 2569 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858985 2569 flags.go:64] FLAG: --v="2" Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858989 2569 flags.go:64] FLAG: --version="false" Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858993 2569 flags.go:64] FLAG: --vmodule="" Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.858997 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.859000 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859124 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859127 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859130 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859134 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859136 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859139 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859142 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859145 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859147 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859149 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859152 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859164 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859167 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859170 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:37.867468 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859173 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859175 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859179 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859183 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859186 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859188 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859209 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859212 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859215 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859217 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859220 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859222 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859225 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859227 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859230 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859232 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859235 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859237 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859240 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:37.868090 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859242 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859245 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859249 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859251 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859254 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859257 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859260 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859262 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859265 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859268 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859272 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859274 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859277 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859279 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859282 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859284 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859287 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859289 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859292 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:37.868548 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859294 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859297 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859299 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859301 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859304 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859306 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859309 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859311 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859314 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859316 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859319 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859321 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859324 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859326 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859329 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859333 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859335 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859338 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859340 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859342 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:37.869017 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859345 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859348 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859350 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859354 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859356 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859359 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859361 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859363 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859366 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859368 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859371 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859373 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859376 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859378 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859380 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859383 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859385 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859388 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859391 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:37.869475 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859393 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859397 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859401 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859404 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859407 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859409 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859411 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859414 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859417 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859420 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859422 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859425 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859427 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859429 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859432 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:37.869926 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.859459 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859500 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859506 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859509 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859512 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859515 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859517 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859520 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859522 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859525 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859527 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859530 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859532 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859535 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859537 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859540 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859542 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859544 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859547 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:37.870275 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859549 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859551 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859554 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859556 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859559 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859561 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859565 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859567 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859570 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859572 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859575 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859578 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859580 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859583 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859585 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859589 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859592 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859595 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859597 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:37.870706 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859600 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859602 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859604 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859607 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859609 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859612 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859614 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859617 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859619 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859622 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859624 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859627 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859630 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859632 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859635 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859637 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859639 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859642 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859644 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859648 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:37.871173 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859650 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859653 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859655 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859658 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859660 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859663 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859665 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859668 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859672 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859675 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859677 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859680 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859683 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859686 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859688 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859691 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859693 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859696 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:37.871667 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859698 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859702 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859706 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859709 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859712 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859714 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859717 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859719 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859721 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859724 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859726 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859729 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859731 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859734 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859737 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859739 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859741 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859744 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859746 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:37.872121 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859749 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859751 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859754 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859758 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859760 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859763 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859765 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859767 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859787 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859790 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859794 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.859796 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.866843 2569 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.866856 2569 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.866868 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.866873 2569 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866946 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866951 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866954 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866957 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:37.872597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866959 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866962 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866964 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866967 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866970 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866973 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866975 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866977 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866980 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866982 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866985 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866987 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866989 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866992 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.866994 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867002 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867005 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867007 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867010 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867012 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:37.873145 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867015 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867017 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867019 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867022 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867024 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867027 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867030 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867032 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867035 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867037 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867040 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867043 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867045 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867047 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867050 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867052 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867055 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867057 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867059 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:37.873597 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867062 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867064 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867067 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867069 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867071 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867073 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867076 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867078 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867080 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867088 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867091 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867093 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867096 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867098 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867100 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867103 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867105 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867107 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867112 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:37.874059 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867116 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867119 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867123 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867126 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867129 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867132 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867134 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867136 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867139 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867141 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867143 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867146 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867148 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867150 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867160 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867163 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867166 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867168 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867170 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867173 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:37.874480 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867175 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867177 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867180 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867188 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867192 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867196 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867198 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867201 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867204 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867206 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867209 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867211 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867214 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867216 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867219 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867221 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867223 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867243 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867246 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:37.874941 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867248 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867251 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867253 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867256 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867258 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.867324 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867538 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867543 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867546 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867549 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867551 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867554 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867556 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867559 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867561 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867564 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:37.875568 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867566 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867574 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867577 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867580 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867582 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867585 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867587 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867590 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867592 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867595 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867597 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867606 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867609 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867611 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867613 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867616 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867618 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867621 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867623 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867625 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:37.876180 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867628 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867630 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867633 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867635 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867637 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867640 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867642 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867645 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867647 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867649 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867652 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867654 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867657 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867659 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867667 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867669 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867671 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867674 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867676 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867678 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:37.876813 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867681 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867683 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867686 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867689 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867691 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867693 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867695 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867698 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867700 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867704 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867707 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867710 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867712 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867715 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867717 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867719 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867722 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867724 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867727 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867729 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:37.877264 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867731 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867734 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867736 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867738 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867741 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867744 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867746 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867754 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867757 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867759 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867761 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867764 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867766 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867785 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867789 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867793 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867795 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867797 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867801 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:37.877711 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867804 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867808 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867810 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867813 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867815 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867818 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867820 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867822 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867825 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867827 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867829 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867832 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867834 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867837 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867839 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867841 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:37.878261 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867844 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.867869 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867914 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867918 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867921 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867929 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867932 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867934 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867937 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867939 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867942 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867944 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867947 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867949 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867952 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867954 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:37.878634 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867957 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867959 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867961 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867964 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867966 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867969 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867971 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867973 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867976 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867978 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867981 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867983 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867985 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867988 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867990 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867994 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.867998 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868003 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868005 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868008 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:37.879021 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868010 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868013 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868021 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868024 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868026 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868029 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868031 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868034 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868036 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868038 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868041 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868043 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868046 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868048 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868050 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868053 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868055 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868057 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868060 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:37.879598 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868062 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868065 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868067 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868070 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868072 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868076 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868079 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868083 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868086 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868088 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868092 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868095 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868098 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868100 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868103 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868106 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868114 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868116 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868119 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:37.880111 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868121 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868123 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868126 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868128 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868131 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868134 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868136 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868138 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868141 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868144 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868148 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868150 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868153 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868155 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868158 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868160 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868163 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868165 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868168 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868170 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:37.880603 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868173 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868175 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868178 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868181 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868184 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868186 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868189 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868191 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868194 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868197 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868205 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868208 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868210 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:37.868213 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.868259 2569 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.870869 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.871964 2569 server.go:1014] "Starting client certificate rotation" Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.872063 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:04:37.881096 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.872103 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:04:37.881564 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.873956 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:04:37.881564 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.877815 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:04:37.888926 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.888909 2569 log.go:25] "Validated CRI v1 runtime API" Nov 03 00:04:37.891766 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.891750 2569 log.go:25] "Validated CRI v1 image API" Nov 03 00:04:37.892579 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.892554 2569 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 00:04:37.898131 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.898108 2569 fs.go:135] Filesystem UUIDs: map[48a685b5-c35f-4506-a580-b8c85d6f1746:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d176709f-55ad-46a9-bce1-bfaa99db4f13:/dev/nvme0n1p3] Nov 03 00:04:37.898195 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.898131 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 00:04:37.904119 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.904011 2569 manager.go:217] Machine: {Timestamp:2026-11-03 00:04:37.90267309 +0000 UTC m=+0.343337109 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3197231 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2263daf80fa0f9e45e9042819d436f SystemUUID:ec2263da-f80f-a0f9-e45e-9042819d436f BootID:97d4bf86-a878-42b9-984d-23e13667ea98 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:03:56:af:06:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:03:56:af:06:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:3a:61:13:0c:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 00:04:37.904119 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.904117 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 00:04:37.904212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.904199 2569 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 00:04:37.906233 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.906210 2569 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 00:04:37.906366 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.906235 2569 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-172.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 00:04:37.906409 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.906369 2569 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 00:04:37.906409 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.906376 2569 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 00:04:37.906409 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.906406 2569 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:04:37.906481 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.906423 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:04:37.907092 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.907082 2569 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 00:04:37.910621 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.910610 2569 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 00:04:37.910738 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.910729 2569 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 00:04:37.912091 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.912078 2569 kubelet.go:493] "Attempting to sync node with API server" Nov 03 00:04:37.912142 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.912102 2569 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 00:04:37.912142 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.912130 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 00:04:37.912194 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.912166 2569 kubelet.go:405] "Adding apiserver pod source" Nov 03 00:04:37.912194 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.912190 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 00:04:37.912533 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.912522 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:04:37.914970 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.914955 2569 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 00:04:37.919278 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.919263 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:04:37.920098 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920084 2569 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 00:04:37.920162 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920124 2569 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 00:04:37.920311 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920298 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 00:04:37.920384 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920319 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 00:04:37.920384 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920329 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 00:04:37.920384 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920338 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 00:04:37.920384 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920346 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 00:04:37.920384 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920355 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 00:04:37.920946 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920936 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 00:04:37.921003 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920965 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 00:04:37.921003 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920981 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 00:04:37.921003 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.920990 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 00:04:37.921106 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.921008 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 00:04:37.921106 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.921021 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 00:04:37.921574 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.921565 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 00:04:37.921620 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.921578 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 00:04:37.925249 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.925234 2569 server.go:1263] "Started kubelet" Nov 03 00:04:37.928321 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.928178 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 00:04:37.928406 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.928361 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 00:04:37.928543 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.925294 2569 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 00:04:37.928977 ip-10-0-140-172 systemd[1]: Started Kubernetes Kubelet. Nov 03 00:04:37.929543 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.929394 2569 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 00:04:37.932203 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.932188 2569 server.go:317] "Adding debug handlers to kubelet server" Nov 03 00:04:37.934217 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:37.934198 2569 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 00:04:37.935513 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.935495 2569 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 00:04:37.935513 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.935509 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 00:04:37.935836 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:37.935818 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-172.us-west-1.compute.internal\" not found" Nov 03 00:04:37.936026 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936008 2569 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 00:04:37.936102 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936030 2569 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 00:04:37.936102 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936071 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 00:04:37.936205 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936134 2569 reconstruct.go:97] "Volume reconstruction finished" Nov 03 00:04:37.936205 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936142 2569 reconciler.go:29] "Reconciler: start to sync state" Nov 03 00:04:37.936561 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936539 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 00:04:37.936561 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936561 2569 factory.go:55] Registering systemd factory Nov 03 00:04:37.936561 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936575 2569 factory.go:223] Registration of the systemd container factory successfully Nov 03 00:04:37.936768 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936755 2569 factory.go:153] Registering CRI-O factory Nov 03 00:04:37.936838 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936790 2569 factory.go:223] Registration of the crio container factory successfully Nov 03 00:04:37.936838 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936805 2569 factory.go:103] Registering Raw factory Nov 03 00:04:37.936838 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.936817 2569 manager.go:1199] Started watching for new ooms in manager Nov 03 00:04:37.937916 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.937901 2569 manager.go:319] Starting recovery of all containers Nov 03 00:04:37.948415 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.948399 2569 manager.go:324] Recovery completed Nov 03 00:04:37.952417 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.952401 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:37.955095 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.954989 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:37.955164 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.955110 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:37.955164 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.955124 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:37.955603 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.955590 2569 cpu_manager.go:226] "Starting" policy="none" Nov 03 00:04:37.955603 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.955601 2569 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 00:04:37.955708 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.955617 2569 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 00:04:37.955708 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.955640 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:37.956312 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.956299 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:37.956376 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.956323 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:37.956376 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.956336 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:37.957946 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.957837 2569 policy_none.go:50] "Start" Nov 03 00:04:37.957946 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.957851 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 00:04:37.957946 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.957860 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 00:04:37.959933 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:37.959922 2569 policy_none.go:44] "Start" Nov 03 00:04:38.010367 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.010337 2569 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 00:04:38.012639 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.012628 2569 manager.go:341] "Starting Device Plugin manager" Nov 03 00:04:38.012729 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:38.012654 2569 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 00:04:38.012729 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.012662 2569 server.go:87] "Starting device plugin registration server" Nov 03 00:04:38.012944 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.012932 2569 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 00:04:38.012993 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.012947 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 00:04:38.013066 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.013051 2569 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 00:04:38.013149 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.013137 2569 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 00:04:38.013195 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.013149 2569 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 00:04:38.013754 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:38.013738 2569 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 00:04:38.013829 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:38.013810 2569 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-172.us-west-1.compute.internal\" not found" Nov 03 00:04:38.064014 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.063975 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 00:04:38.065906 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.065302 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 00:04:38.065906 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.065319 2569 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 00:04:38.065906 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.065345 2569 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 00:04:38.065906 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:38.065376 2569 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 00:04:38.068244 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.068223 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-172.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 00:04:38.068244 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:38.068220 2569 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-172.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 00:04:38.113461 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.113389 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.120875 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.120851 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.120875 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:38.119590 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-172.us-west-1.compute.internal.18e460285409e54a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-172.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-172.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 00:04:37.925193034 +0000 UTC m=+0.365857050,LastTimestamp:2026-11-03 00:04:37.925193034 +0000 UTC m=+0.365857050,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-172.us-west-1.compute.internal,}" Nov 03 00:04:38.121061 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.120884 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.121061 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.120901 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.121061 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.120933 2569 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.146970 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.146944 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 00:04:38.147085 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.147011 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dtd8r" Nov 03 00:04:38.166032 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.166009 2569 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal"] Nov 03 00:04:38.166112 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.166061 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.166911 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.166897 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.166996 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.166922 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.166996 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.166932 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.169140 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.169129 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.169284 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.169270 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.169347 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.169306 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.169871 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.169856 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.169871 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.169867 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.169997 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.169880 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.169997 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.169891 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.169997 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.169903 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.169997 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.169893 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.171960 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.171943 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.172026 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.171976 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.172644 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.172630 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.172702 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.172658 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.172702 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.172670 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.205446 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:38.205426 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-172.us-west-1.compute.internal\" not found" node="ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.208045 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:38.208029 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-172.us-west-1.compute.internal\" not found" node="ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.223961 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.223945 2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:38.225662 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.225646 2569 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:38.238867 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.238845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.238961 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.238875 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.238961 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.238903 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a634ffde00653c750ef568ec11d5617-config\") pod \"kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"3a634ffde00653c750ef568ec11d5617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.323125 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.323087 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:38.324240 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.324222 2569 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:38.325581 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.325388 2569 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.325581 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:38.325423 2569 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-172.us-west-1.compute.internal\": node \"ip-10-0-140-172.us-west-1.compute.internal\" not found" Nov 03 00:04:38.335686 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.335663 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.339230 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.339193 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.339334 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.339237 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a634ffde00653c750ef568ec11d5617-config\") pod \"kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"3a634ffde00653c750ef568ec11d5617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.339334 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.339258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.339334 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.339304 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.339334 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.339306 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.339334 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.339308 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a634ffde00653c750ef568ec11d5617-config\") pod \"kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"3a634ffde00653c750ef568ec11d5617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.391561 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.391506 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:04:38.392130 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.392118 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.450351 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.450329 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:04:38.508086 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.508060 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.510964 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.510936 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:04:38.871358 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.871286 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 00:04:38.871941 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.871440 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 00:04:38.871941 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.871464 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 00:04:38.871941 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:38.871418 2569 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a94de1dd7bb8b42158c74d34dda92279-9029fc03a8717821.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-172.us-west-1.compute.internal.18e4602855d22905\": read tcp 10.0.140.172:54852->18.189.122.36:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-172.us-west-1.compute.internal.18e4602855d22905 default 5109 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-172.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-172.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-172.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 00:04:37 +0000 UTC,LastTimestamp:2026-11-03 00:04:38.166909828 +0000 UTC m=+0.607573844,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-172.us-west-1.compute.internal,}" Nov 03 00:04:38.871941 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.871491 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 00:04:38.912603 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.912584 2569 apiserver.go:52] "Watching apiserver" Nov 03 00:04:38.936273 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:38.936254 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 00:04:39.151113 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.151040 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:38 +0000 UTC" deadline="2028-08-03 08:07:46.0490992 +0000 UTC" Nov 03 00:04:39.151113 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.151066 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15344h3m6.898035837s" Nov 03 00:04:39.174951 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.174930 2569 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 00:04:39.175283 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.175264 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-dz9d5","kube-system/konnectivity-agent-rlgnn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj","openshift-dns/node-resolver-lk69m","openshift-multus/multus-mmcgw","openshift-ovn-kubernetes/ovnkube-node-cwtgz","kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-d9dlt","openshift-image-registry/node-ca-nk4xm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-kx7kh","openshift-multus/network-metrics-daemon-zvfng","openshift-network-diagnostics/network-check-target-bthfh","openshift-network-operator/iptables-alerter-rc4t7"] Nov 03 00:04:39.177769 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.177751 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.179722 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.179703 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:04:39.181754 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.181740 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.183789 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.183753 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lk69m" Nov 03 00:04:39.185829 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.185809 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.188026 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.188011 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.190169 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.190152 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:04:39.192297 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.192269 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:39.192385 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.192364 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:04:39.194960 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.194929 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:04:39.197730 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.197715 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:39.197814 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.197785 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:04:39.200581 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.200566 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.204138 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.204121 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:39.204200 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.204178 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:04:39.237255 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.237236 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 00:04:39.237565 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.237544 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 00:04:39.237662 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.237611 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 00:04:39.237662 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.237629 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.237792 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.237662 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.237792 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.237692 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 00:04:39.237792 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.237703 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 00:04:39.237792 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.237730 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 00:04:39.237792 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.237760 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7949\"" Nov 03 00:04:39.237792 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.237763 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pkcxp\"" Nov 03 00:04:39.239989 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.239975 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.240142 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.240128 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z6zhf\"" Nov 03 00:04:39.240142 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.240135 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 00:04:39.240228 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.240136 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.243256 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243240 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:39.243332 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243265 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq7js\" (UniqueName: \"kubernetes.io/projected/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-kube-api-access-qq7js\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.243332 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243280 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-dbus\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:39.243332 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243297 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-system-cni-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.243427 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-registration-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.243427 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243383 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-lib-modules\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.243427 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243411 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/434003bf-315d-42b3-8ee4-9cdc53936d73-konnectivity-ca\") pod \"konnectivity-agent-rlgnn\" (UID: \"434003bf-315d-42b3-8ee4-9cdc53936d73\") " pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:04:39.243514 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-var-lib-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.243514 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243468 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa475895-c30a-4076-98c0-d941610c151f-serviceca\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:04:39.243514 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243497 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-kubernetes\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.243597 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243525 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-multus-certs\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.243597 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243543 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.243597 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243572 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.243679 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-hosts-file\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:04:39.243679 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243639 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-tmp-dir\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:04:39.243679 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243656 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-log-socket\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.243760 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243698 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed2418f7-c460-4057-bbc1-9d5f01278635-ovn-node-metrics-cert\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.243760 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243715 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-host\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.243760 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243731 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cni-binary-copy\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.243760 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243744 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-systemd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.243760 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243758 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-bin\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.243917 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243800 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.243917 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243825 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-bin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.243917 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243841 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-multus\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.243917 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243866 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-conf-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.243917 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243889 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4nfn\" (UniqueName: \"kubernetes.io/projected/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-kube-api-access-h4nfn\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.243917 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnq68\" (UniqueName: \"kubernetes.io/projected/e744436f-8167-45f8-8e07-508d1cc24f27-kube-api-access-fnq68\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:04:39.244078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243920 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.244078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243936 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-conf\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.244078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243965 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-tuned\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.244078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.243985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-socket-dir-parent\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.244078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244005 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-cnibin\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.244078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244025 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-slash\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.244078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244039 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-ovn\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.244078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244052 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.244078 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244066 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjs8\" (UniqueName: \"kubernetes.io/projected/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kube-api-access-7kjs8\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.244408 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244079 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa475895-c30a-4076-98c0-d941610c151f-host\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:04:39.244408 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244137 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-modprobe-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.244408 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.244408 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244204 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqv6\" (UniqueName: \"kubernetes.io/projected/fa475895-c30a-4076-98c0-d941610c151f-kube-api-access-glqv6\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:04:39.244408 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-netns\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.244408 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244258 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.244408 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244273 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-netns\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.244408 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244318 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-node-log\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.244408 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244347 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-config\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.244408 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244371 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-tmp\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.244408 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244429 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-hostroot\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244451 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-daemon-config\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244487 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rwgs\" (UniqueName: \"kubernetes.io/projected/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-kube-api-access-4rwgs\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244509 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244525 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e744436f-8167-45f8-8e07-508d1cc24f27-iptables-alerter-script\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244547 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e744436f-8167-45f8-8e07-508d1cc24f27-host-slash\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244567 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-kubelet\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244603 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-systemd\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244630 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-kubelet-config\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244663 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-env-overrides\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ctz\" (UniqueName: \"kubernetes.io/projected/ed2418f7-c460-4057-bbc1-9d5f01278635-kube-api-access-m4ctz\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244705 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwlj\" (UniqueName: \"kubernetes.io/projected/825e190f-db64-4823-8b20-4b605ba2c336-kube-api-access-vhwlj\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244732 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-run\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.244787 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244753 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/434003bf-315d-42b3-8ee4-9cdc53936d73-agent-certs\") pod \"konnectivity-agent-rlgnn\" (UID: \"434003bf-315d-42b3-8ee4-9cdc53936d73\") " pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244802 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-netd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244817 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-script-lib\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244839 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-system-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244855 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-etc-kubernetes\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244871 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br9kn\" (UniqueName: \"kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn\") pod \"network-check-target-bthfh\" (UID: \"9593d56b-b401-4c0b-80cc-21a9e034b75d\") " pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysconfig\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244909 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244924 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4dj\" (UniqueName: \"kubernetes.io/projected/47d5d066-c799-4be0-9a2c-680351cb8095-kube-api-access-lk4dj\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-socket-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244956 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-sys-fs\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244968 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-etc-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244981 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-var-lib-kubelet\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.244994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-k8s-cni-cncf-io\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.245025 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.245065 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-device-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.245315 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.245099 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-systemd-units\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.245719 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.245128 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-sys\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.245719 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.245157 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cnibin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.245719 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.245180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-kubelet\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.332088 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.332065 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 00:04:39.332185 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.332110 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dqd6q" Nov 03 00:04:39.337885 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.337860 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.340352 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.338579 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.340352 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.338586 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f67qw\"" Nov 03 00:04:39.340352 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.338634 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.340352 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.338680 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.340352 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.338800 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.340352 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.338880 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 00:04:39.340352 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.339007 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.340352 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.339030 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 00:04:39.340352 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.339104 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 00:04:39.340352 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.339242 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ptgp6\"" Nov 03 00:04:39.345430 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345412 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-bin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.345518 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-multus\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.345518 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345466 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-conf-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.345518 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345489 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nfn\" (UniqueName: \"kubernetes.io/projected/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-kube-api-access-h4nfn\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.345518 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345512 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnq68\" (UniqueName: \"kubernetes.io/projected/e744436f-8167-45f8-8e07-508d1cc24f27-kube-api-access-fnq68\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:04:39.345706 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345519 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-multus\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.345706 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345537 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.345706 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345525 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-conf-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.345706 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345491 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-bin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.345706 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-conf\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.345706 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345588 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.345706 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-tuned\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.345706 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-socket-dir-parent\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.345706 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345656 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-cnibin\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.345706 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-slash\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.345706 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345703 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-ovn\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345727 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345751 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-slash\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345754 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-cnibin\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345752 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjs8\" (UniqueName: \"kubernetes.io/projected/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kube-api-access-7kjs8\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345788 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-ovn\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345702 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-conf\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345815 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-socket-dir-parent\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345848 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa475895-c30a-4076-98c0-d941610c151f-host\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345852 2569 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345886 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-modprobe-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345894 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345906 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345939 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa475895-c30a-4076-98c0-d941610c151f-host\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqv6\" (UniqueName: \"kubernetes.io/projected/fa475895-c30a-4076-98c0-d941610c151f-kube-api-access-glqv6\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.345999 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-netns\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346022 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346057 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-modprobe-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.346212 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346078 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-netns\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-netns\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346128 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346132 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-node-log\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346157 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-config\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346163 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-node-log\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346144 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-netns\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346183 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-tmp\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-hostroot\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-daemon-config\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346263 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwgs\" (UniqueName: \"kubernetes.io/projected/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-kube-api-access-4rwgs\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346283 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-hostroot\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e744436f-8167-45f8-8e07-508d1cc24f27-iptables-alerter-script\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.346349 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346377 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e744436f-8167-45f8-8e07-508d1cc24f27-host-slash\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:04:39.347087 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.346437 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs podName:825e190f-db64-4823-8b20-4b605ba2c336 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:39.846420684 +0000 UTC m=+2.287084704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs") pod "network-metrics-daemon-zvfng" (UID: "825e190f-db64-4823-8b20-4b605ba2c336") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346349 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e744436f-8167-45f8-8e07-508d1cc24f27-host-slash\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346468 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-kubelet\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346475 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346488 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346506 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-systemd\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346526 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-kubelet-config\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-env-overrides\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346561 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346568 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ctz\" (UniqueName: \"kubernetes.io/projected/ed2418f7-c460-4057-bbc1-9d5f01278635-kube-api-access-m4ctz\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346586 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346590 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwlj\" (UniqueName: \"kubernetes.io/projected/825e190f-db64-4823-8b20-4b605ba2c336-kube-api-access-vhwlj\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346545 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-kubelet\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-run\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346663 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-config\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346669 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-systemd\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346664 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/434003bf-315d-42b3-8ee4-9cdc53936d73-agent-certs\") pod \"konnectivity-agent-rlgnn\" (UID: \"434003bf-315d-42b3-8ee4-9cdc53936d73\") " pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:04:39.348097 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-netd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346704 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-kubelet-config\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-script-lib\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346744 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-netd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346753 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-system-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-etc-kubernetes\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346806 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-run\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346831 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9kn\" (UniqueName: \"kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn\") pod \"network-check-target-bthfh\" (UID: \"9593d56b-b401-4c0b-80cc-21a9e034b75d\") " pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346854 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysconfig\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346858 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-system-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346904 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysconfig\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.346999 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-etc-kubernetes\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347032 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347054 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4dj\" (UniqueName: \"kubernetes.io/projected/47d5d066-c799-4be0-9a2c-680351cb8095-kube-api-access-lk4dj\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-socket-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347098 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-sys-fs\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347121 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-etc-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.348873 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-var-lib-kubelet\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-k8s-cni-cncf-io\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347178 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347213 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347231 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-socket-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-device-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347246 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-etc-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347260 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-systemd-units\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347299 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-sys-fs\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347308 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-k8s-cni-cncf-io\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347353 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-device-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347402 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-var-lib-kubelet\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347436 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-sys\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347446 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-systemd-units\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347469 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cnibin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347498 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-sys\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-kubelet\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.349968 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347549 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cnibin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7js\" (UniqueName: \"kubernetes.io/projected/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-kube-api-access-qq7js\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347580 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-kubelet\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.347618 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.347664 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret podName:ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad nodeName:}" failed. No retries permitted until 2026-11-03 00:04:39.847649773 +0000 UTC m=+2.288313777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret") pod "global-pull-secret-syncer-dz9d5" (UID: "ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347679 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-dbus\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347696 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-system-cni-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347711 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-registration-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347731 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-lib-modules\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347755 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-system-cni-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347756 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/434003bf-315d-42b3-8ee4-9cdc53936d73-konnectivity-ca\") pod \"konnectivity-agent-rlgnn\" (UID: \"434003bf-315d-42b3-8ee4-9cdc53936d73\") " pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347793 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-registration-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347811 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-var-lib-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347835 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa475895-c30a-4076-98c0-d941610c151f-serviceca\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347849 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-dbus\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347854 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-kubernetes\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.350452 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347878 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-kubernetes\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347882 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-lib-modules\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347890 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-multus-certs\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347902 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-var-lib-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347926 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-multus-certs\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347945 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347969 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-hosts-file\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.347990 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-tmp-dir\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-log-socket\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348025 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed2418f7-c460-4057-bbc1-9d5f01278635-ovn-node-metrics-cert\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348066 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-host\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348089 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cni-binary-copy\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348108 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-systemd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-bin\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348150 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.350910 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348216 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348252 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-log-socket\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348278 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-tmp-dir\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348356 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-hosts-file\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348362 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348414 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-host\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348454 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-systemd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348492 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-bin\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348491 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.348800 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cni-binary-copy\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.349854 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-daemon-config\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.349977 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-tuned\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.350024 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-tmp\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.350201 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/434003bf-315d-42b3-8ee4-9cdc53936d73-agent-certs\") pod \"konnectivity-agent-rlgnn\" (UID: \"434003bf-315d-42b3-8ee4-9cdc53936d73\") " pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.350247 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/434003bf-315d-42b3-8ee4-9cdc53936d73-konnectivity-ca\") pod \"konnectivity-agent-rlgnn\" (UID: \"434003bf-315d-42b3-8ee4-9cdc53936d73\") " pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:04:39.351373 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.350407 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed2418f7-c460-4057-bbc1-9d5f01278635-ovn-node-metrics-cert\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.354450 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354437 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 00:04:39.354549 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354534 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.354549 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354546 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.354662 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354550 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.354662 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354582 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 00:04:39.354662 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354600 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k6628\"" Nov 03 00:04:39.354846 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354827 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 00:04:39.354929 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354857 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scmmg\"" Nov 03 00:04:39.354929 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354899 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8glgl\"" Nov 03 00:04:39.354929 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354916 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nhzpn\"" Nov 03 00:04:39.354929 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354925 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 00:04:39.355052 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.354917 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.356904 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.356890 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e744436f-8167-45f8-8e07-508d1cc24f27-iptables-alerter-script\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:04:39.357531 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.357515 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-env-overrides\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.357691 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.357675 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-script-lib\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.358613 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.358596 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa475895-c30a-4076-98c0-d941610c151f-serviceca\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:04:39.401959 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.401605 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:39.401959 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.401628 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:39.401959 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.401640 2569 projected.go:196] Error preparing data for projected volume kube-api-access-br9kn for pod openshift-network-diagnostics/network-check-target-bthfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:39.401959 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.401698 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn podName:9593d56b-b401-4c0b-80cc-21a9e034b75d nodeName:}" failed. No retries permitted until 2026-11-03 00:04:39.901678447 +0000 UTC m=+2.342342453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-br9kn" (UniqueName: "kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn") pod "network-check-target-bthfh" (UID: "9593d56b-b401-4c0b-80cc-21a9e034b75d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:39.403831 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.403806 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnq68\" (UniqueName: \"kubernetes.io/projected/e744436f-8167-45f8-8e07-508d1cc24f27-kube-api-access-fnq68\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:04:39.403931 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.403836 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjs8\" (UniqueName: \"kubernetes.io/projected/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kube-api-access-7kjs8\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.403931 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.403832 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7js\" (UniqueName: \"kubernetes.io/projected/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-kube-api-access-qq7js\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.404650 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.404634 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nfn\" (UniqueName: \"kubernetes.io/projected/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-kube-api-access-h4nfn\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.404923 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.404905 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rwgs\" (UniqueName: \"kubernetes.io/projected/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-kube-api-access-4rwgs\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:04:39.404993 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.404978 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqv6\" (UniqueName: \"kubernetes.io/projected/fa475895-c30a-4076-98c0-d941610c151f-kube-api-access-glqv6\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:04:39.405142 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.405121 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4dj\" (UniqueName: \"kubernetes.io/projected/47d5d066-c799-4be0-9a2c-680351cb8095-kube-api-access-lk4dj\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.405621 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.405604 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ctz\" (UniqueName: \"kubernetes.io/projected/ed2418f7-c460-4057-bbc1-9d5f01278635-kube-api-access-m4ctz\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.406063 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.406050 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwlj\" (UniqueName: \"kubernetes.io/projected/825e190f-db64-4823-8b20-4b605ba2c336-kube-api-access-vhwlj\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:39.512161 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.512117 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:04:39.516338 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.516298 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:04:39.531838 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.531819 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:04:39.541657 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.541638 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lk69m" Nov 03 00:04:39.545186 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.545147 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mmcgw" Nov 03 00:04:39.589938 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.589906 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:04:39.594566 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.594547 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:04:39.613123 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.613104 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:04:39.617850 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.617835 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:04:39.649486 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:39.649449 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bb3e5f_3afc_4201_9ae7_3b3e8b0c27a0.slice/crio-302c00ce1e797a5fca493e1461c36f4cf7f665ef102ee649c93d8dab7a932dce WatchSource:0}: Error finding container 302c00ce1e797a5fca493e1461c36f4cf7f665ef102ee649c93d8dab7a932dce: Status 404 returned error can't find the container with id 302c00ce1e797a5fca493e1461c36f4cf7f665ef102ee649c93d8dab7a932dce Nov 03 00:04:39.650292 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:39.650271 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20bb628721d473930b8a876f613ab5c.slice/crio-18f04304a2355eb8ec06ee6f09511152a167e79e98b977ac62db21d88e394809 WatchSource:0}: Error finding container 18f04304a2355eb8ec06ee6f09511152a167e79e98b977ac62db21d88e394809: Status 404 returned error can't find the container with id 18f04304a2355eb8ec06ee6f09511152a167e79e98b977ac62db21d88e394809 Nov 03 00:04:39.651588 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:39.651569 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a634ffde00653c750ef568ec11d5617.slice/crio-d39c93f03caa50f35b40e92aedf5376c8cf5911d789f150e8cfaa2e77af45bd0 WatchSource:0}: Error finding container d39c93f03caa50f35b40e92aedf5376c8cf5911d789f150e8cfaa2e77af45bd0: Status 404 returned error can't find the container with id d39c93f03caa50f35b40e92aedf5376c8cf5911d789f150e8cfaa2e77af45bd0 Nov 03 00:04:39.655940 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.655923 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:04:39.745646 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.745622 2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:39.852133 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.852097 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:39.852330 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.852147 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:39.852330 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.852245 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:39.852330 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.852254 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:39.852330 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.852297 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret podName:ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad nodeName:}" failed. No retries permitted until 2026-11-03 00:04:40.852282323 +0000 UTC m=+3.292946328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret") pod "global-pull-secret-syncer-dz9d5" (UID: "ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:39.852330 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.852310 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs podName:825e190f-db64-4823-8b20-4b605ba2c336 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:40.852303977 +0000 UTC m=+3.292967982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs") pod "network-metrics-daemon-zvfng" (UID: "825e190f-db64-4823-8b20-4b605ba2c336") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:39.868177 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.868139 2569 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:39.922357 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.922279 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:39.952953 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:39.952918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9kn\" (UniqueName: \"kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn\") pod \"network-check-target-bthfh\" (UID: \"9593d56b-b401-4c0b-80cc-21a9e034b75d\") " pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:39.953113 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.953095 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:39.953158 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.953116 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:39.953158 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.953127 2569 projected.go:196] Error preparing data for projected volume kube-api-access-br9kn for pod openshift-network-diagnostics/network-check-target-bthfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:39.953245 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:39.953178 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn podName:9593d56b-b401-4c0b-80cc-21a9e034b75d nodeName:}" failed. No retries permitted until 2026-11-03 00:04:40.953163446 +0000 UTC m=+3.393827451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-br9kn" (UniqueName: "kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn") pod "network-check-target-bthfh" (UID: "9593d56b-b401-4c0b-80cc-21a9e034b75d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:40.069937 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:40.069877 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" event={"ID":"3a634ffde00653c750ef568ec11d5617","Type":"ContainerStarted","Data":"d39c93f03caa50f35b40e92aedf5376c8cf5911d789f150e8cfaa2e77af45bd0"} Nov 03 00:04:40.071023 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:40.071000 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" event={"ID":"d20bb628721d473930b8a876f613ab5c","Type":"ContainerStarted","Data":"18f04304a2355eb8ec06ee6f09511152a167e79e98b977ac62db21d88e394809"} Nov 03 00:04:40.072040 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:40.072020 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" event={"ID":"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0","Type":"ContainerStarted","Data":"302c00ce1e797a5fca493e1461c36f4cf7f665ef102ee649c93d8dab7a932dce"} Nov 03 00:04:40.337737 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:40.337615 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:39 +0000 UTC" deadline="2028-07-02 20:42:50.303717053 +0000 UTC" Nov 03 00:04:40.337737 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:40.337699 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14588h38m9.966028123s" Nov 03 00:04:40.362440 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:40.362412 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d5d066_c799_4be0_9a2c_680351cb8095.slice/crio-cfcd096023b1bf068a242ef678870dd8f889f5cdfeb17ed1944ce715a52c1f1f WatchSource:0}: Error finding container cfcd096023b1bf068a242ef678870dd8f889f5cdfeb17ed1944ce715a52c1f1f: Status 404 returned error can't find the container with id cfcd096023b1bf068a242ef678870dd8f889f5cdfeb17ed1944ce715a52c1f1f Nov 03 00:04:40.436662 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:40.436634 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a058bf9_45a2_44fd_908f_0e8fc41c1e5b.slice/crio-7d476c7e47d2c99c05df779167116f85636f440efdc4f4ac5bcb5fb20c3f7c39 WatchSource:0}: Error finding container 7d476c7e47d2c99c05df779167116f85636f440efdc4f4ac5bcb5fb20c3f7c39: Status 404 returned error can't find the container with id 7d476c7e47d2c99c05df779167116f85636f440efdc4f4ac5bcb5fb20c3f7c39 Nov 03 00:04:40.480074 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:40.480047 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc26ec7_d383_4580_a5c4_becaf9dc3278.slice/crio-46198f6a87e25b1adf1568ca64c975901243d2a1d33d4a81d8ed313568007dd1 WatchSource:0}: Error finding container 46198f6a87e25b1adf1568ca64c975901243d2a1d33d4a81d8ed313568007dd1: Status 404 returned error can't find the container with id 46198f6a87e25b1adf1568ca64c975901243d2a1d33d4a81d8ed313568007dd1 Nov 03 00:04:40.480850 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:40.480814 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa475895_c30a_4076_98c0_d941610c151f.slice/crio-add431a4b4a0904c81c9808cc3b4724f9759184f64b2854c4bebb7f78e1f33e5 WatchSource:0}: Error finding container add431a4b4a0904c81c9808cc3b4724f9759184f64b2854c4bebb7f78e1f33e5: Status 404 returned error can't find the container with id add431a4b4a0904c81c9808cc3b4724f9759184f64b2854c4bebb7f78e1f33e5 Nov 03 00:04:40.523854 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:40.523801 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434003bf_315d_42b3_8ee4_9cdc53936d73.slice/crio-b4533793b83e46b1869964c16b94011bc06c00ea9a0d05afd66cdd4b17eeb784 WatchSource:0}: Error finding container b4533793b83e46b1869964c16b94011bc06c00ea9a0d05afd66cdd4b17eeb784: Status 404 returned error can't find the container with id b4533793b83e46b1869964c16b94011bc06c00ea9a0d05afd66cdd4b17eeb784 Nov 03 00:04:40.532005 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:40.531980 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode744436f_8167_45f8_8e07_508d1cc24f27.slice/crio-b422b0aac24babb59aa37e41db436ac16ae2c9076b0c6986f70fbf2a15cb3bd3 WatchSource:0}: Error finding container b422b0aac24babb59aa37e41db436ac16ae2c9076b0c6986f70fbf2a15cb3bd3: Status 404 returned error can't find the container with id b422b0aac24babb59aa37e41db436ac16ae2c9076b0c6986f70fbf2a15cb3bd3 Nov 03 00:04:40.539833 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:40.539811 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fd5ced_e9c1_4b9a_9939_473318ebbd93.slice/crio-d19b52601e459e2ea430b7723b2510275d128ca9d96c89f9a6003efa6a937288 WatchSource:0}: Error finding container d19b52601e459e2ea430b7723b2510275d128ca9d96c89f9a6003efa6a937288: Status 404 returned error can't find the container with id d19b52601e459e2ea430b7723b2510275d128ca9d96c89f9a6003efa6a937288 Nov 03 00:04:40.550101 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:04:40.550074 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2418f7_c460_4057_bbc1_9d5f01278635.slice/crio-40fafab6756a07481e1c69da53921402df85d8479a7222b64282dcce0e456272 WatchSource:0}: Error finding container 40fafab6756a07481e1c69da53921402df85d8479a7222b64282dcce0e456272: Status 404 returned error can't find the container with id 40fafab6756a07481e1c69da53921402df85d8479a7222b64282dcce0e456272 Nov 03 00:04:40.860810 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:40.860760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:40.860984 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:40.860850 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:40.860984 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:40.860919 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:40.861134 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:40.861003 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs podName:825e190f-db64-4823-8b20-4b605ba2c336 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:42.860981678 +0000 UTC m=+5.301645688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs") pod "network-metrics-daemon-zvfng" (UID: "825e190f-db64-4823-8b20-4b605ba2c336") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:40.861134 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:40.861020 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:40.861134 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:40.861118 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret podName:ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad nodeName:}" failed. No retries permitted until 2026-11-03 00:04:42.861101691 +0000 UTC m=+5.301765718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret") pod "global-pull-secret-syncer-dz9d5" (UID: "ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:40.962289 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:40.962250 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9kn\" (UniqueName: \"kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn\") pod \"network-check-target-bthfh\" (UID: \"9593d56b-b401-4c0b-80cc-21a9e034b75d\") " pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:40.962969 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:40.962462 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:40.962969 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:40.962496 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:40.962969 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:40.962529 2569 projected.go:196] Error preparing data for projected volume kube-api-access-br9kn for pod openshift-network-diagnostics/network-check-target-bthfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:40.962969 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:40.962591 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn podName:9593d56b-b401-4c0b-80cc-21a9e034b75d nodeName:}" failed. No retries permitted until 2026-11-03 00:04:42.962571231 +0000 UTC m=+5.403235245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-br9kn" (UniqueName: "kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn") pod "network-check-target-bthfh" (UID: "9593d56b-b401-4c0b-80cc-21a9e034b75d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:41.065791 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.065753 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:41.065998 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:41.065917 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:04:41.066073 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.066001 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:41.066116 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:41.066077 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:04:41.066152 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.066140 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:41.066229 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:41.066210 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:04:41.077238 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.077193 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rc4t7" event={"ID":"e744436f-8167-45f8-8e07-508d1cc24f27","Type":"ContainerStarted","Data":"b422b0aac24babb59aa37e41db436ac16ae2c9076b0c6986f70fbf2a15cb3bd3"} Nov 03 00:04:41.078906 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.078859 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lk69m" event={"ID":"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b","Type":"ContainerStarted","Data":"7d476c7e47d2c99c05df779167116f85636f440efdc4f4ac5bcb5fb20c3f7c39"} Nov 03 00:04:41.080438 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.080413 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7kh" event={"ID":"47d5d066-c799-4be0-9a2c-680351cb8095","Type":"ContainerStarted","Data":"cfcd096023b1bf068a242ef678870dd8f889f5cdfeb17ed1944ce715a52c1f1f"} Nov 03 00:04:41.082008 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.081984 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" event={"ID":"ed2418f7-c460-4057-bbc1-9d5f01278635","Type":"ContainerStarted","Data":"40fafab6756a07481e1c69da53921402df85d8479a7222b64282dcce0e456272"} Nov 03 00:04:41.083456 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.083418 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mmcgw" event={"ID":"b3fd5ced-e9c1-4b9a-9939-473318ebbd93","Type":"ContainerStarted","Data":"d19b52601e459e2ea430b7723b2510275d128ca9d96c89f9a6003efa6a937288"} Nov 03 00:04:41.085794 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.084876 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rlgnn" event={"ID":"434003bf-315d-42b3-8ee4-9cdc53936d73","Type":"ContainerStarted","Data":"b4533793b83e46b1869964c16b94011bc06c00ea9a0d05afd66cdd4b17eeb784"} Nov 03 00:04:41.086739 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.086710 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nk4xm" event={"ID":"fa475895-c30a-4076-98c0-d941610c151f","Type":"ContainerStarted","Data":"add431a4b4a0904c81c9808cc3b4724f9759184f64b2854c4bebb7f78e1f33e5"} Nov 03 00:04:41.088825 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.088802 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" event={"ID":"8bc26ec7-d383-4580-a5c4-becaf9dc3278","Type":"ContainerStarted","Data":"46198f6a87e25b1adf1568ca64c975901243d2a1d33d4a81d8ed313568007dd1"} Nov 03 00:04:41.338939 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.338845 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:39 +0000 UTC" deadline="2028-08-08 02:02:51.791061084 +0000 UTC" Nov 03 00:04:41.338939 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:41.338894 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15457h58m10.452173444s" Nov 03 00:04:42.880400 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:42.880357 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:42.880913 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:42.880446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:42.880913 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:42.880545 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:42.880913 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:42.880557 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:42.880913 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:42.880626 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs podName:825e190f-db64-4823-8b20-4b605ba2c336 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:46.880604684 +0000 UTC m=+9.321268693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs") pod "network-metrics-daemon-zvfng" (UID: "825e190f-db64-4823-8b20-4b605ba2c336") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:42.880913 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:42.880647 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret podName:ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad nodeName:}" failed. No retries permitted until 2026-11-03 00:04:46.880636955 +0000 UTC m=+9.321300962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret") pod "global-pull-secret-syncer-dz9d5" (UID: "ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:42.981224 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:42.980872 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9kn\" (UniqueName: \"kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn\") pod \"network-check-target-bthfh\" (UID: \"9593d56b-b401-4c0b-80cc-21a9e034b75d\") " pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:42.981224 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:42.981096 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:42.981224 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:42.981122 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:42.981224 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:42.981135 2569 projected.go:196] Error preparing data for projected volume kube-api-access-br9kn for pod openshift-network-diagnostics/network-check-target-bthfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:42.981224 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:42.981193 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn podName:9593d56b-b401-4c0b-80cc-21a9e034b75d nodeName:}" failed. No retries permitted until 2026-11-03 00:04:46.981175797 +0000 UTC m=+9.421839805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-br9kn" (UniqueName: "kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn") pod "network-check-target-bthfh" (UID: "9593d56b-b401-4c0b-80cc-21a9e034b75d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:43.066508 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:43.066368 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:43.066508 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:43.066498 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:04:43.066740 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:43.066581 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:43.066740 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:43.066667 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:04:43.066740 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:43.066721 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:43.066930 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:43.066784 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:04:44.118889 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:44.118540 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" event={"ID":"3a634ffde00653c750ef568ec11d5617","Type":"ContainerStarted","Data":"22d0576dc690dd9691f0fa5758c339ea8d9e00e2816c698a8c13cbe313e364ab"} Nov 03 00:04:45.065630 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:45.065593 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:45.065848 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:45.065741 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:04:45.065848 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:45.065826 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:45.065983 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:45.065906 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:04:45.065983 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:45.065968 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:45.066085 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:45.066054 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:04:45.140623 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:45.139692 2569 generic.go:356] "Generic (PLEG): container finished" podID="d20bb628721d473930b8a876f613ab5c" containerID="ed299c83332a5167d8279eb390f4bfd629ab5979bf63f21c3153690d3485f7d4" exitCode=0 Nov 03 00:04:45.140623 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:45.139809 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" event={"ID":"d20bb628721d473930b8a876f613ab5c","Type":"ContainerDied","Data":"ed299c83332a5167d8279eb390f4bfd629ab5979bf63f21c3153690d3485f7d4"} Nov 03 00:04:45.147923 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:45.147278 2569 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="500a1a8a36c8c959db627e2aa5e363272f1c1b9ac70bc971ed293776f40ad558" exitCode=0 Nov 03 00:04:45.147923 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:45.147385 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7kh" event={"ID":"47d5d066-c799-4be0-9a2c-680351cb8095","Type":"ContainerDied","Data":"500a1a8a36c8c959db627e2aa5e363272f1c1b9ac70bc971ed293776f40ad558"} Nov 03 00:04:45.267321 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:45.266129 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" podStartSLOduration=6.266108415 podStartE2EDuration="6.266108415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:44.239215292 +0000 UTC m=+6.679879341" watchObservedRunningTime="2026-11-03 00:04:45.266108415 +0000 UTC m=+7.706772446" Nov 03 00:04:46.156093 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:46.155962 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" event={"ID":"d20bb628721d473930b8a876f613ab5c","Type":"ContainerStarted","Data":"9b0974df247669d56cbee9c607370d2ba8ed1d6b5ce290d5c1d2fdf98b5ae376"} Nov 03 00:04:46.159027 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:46.158990 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rlgnn" event={"ID":"434003bf-315d-42b3-8ee4-9cdc53936d73","Type":"ContainerStarted","Data":"cc44dd4d630a5611a1a1bf20e368be49ff29f6aa6dc8a2027dc39d915bbd8ec7"} Nov 03 00:04:46.278693 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:46.278622 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" podStartSLOduration=7.278600832 podStartE2EDuration="7.278600832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:46.27773343 +0000 UTC m=+8.718397460" watchObservedRunningTime="2026-11-03 00:04:46.278600832 +0000 UTC m=+8.719264861" Nov 03 00:04:46.400671 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:46.400607 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rlgnn" podStartSLOduration=3.115635954 podStartE2EDuration="8.40058515s" totalImagesPullingTime="5.284949196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:46.399920445 +0000 UTC m=+8.840584475" watchObservedRunningTime="2026-11-03 00:04:46.40058515 +0000 UTC m=+8.841249182" Nov 03 00:04:46.921687 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:46.921151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:46.921687 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:46.921255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:46.921687 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:46.921422 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:46.921687 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:46.921504 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret podName:ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad nodeName:}" failed. No retries permitted until 2026-11-03 00:04:54.921482442 +0000 UTC m=+17.362146449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret") pod "global-pull-secret-syncer-dz9d5" (UID: "ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:46.921687 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:46.921572 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:46.921687 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:46.921608 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs podName:825e190f-db64-4823-8b20-4b605ba2c336 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:54.921596552 +0000 UTC m=+17.362260563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs") pod "network-metrics-daemon-zvfng" (UID: "825e190f-db64-4823-8b20-4b605ba2c336") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:46.936960 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:46.936771 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:04:46.937565 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:46.937350 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:04:47.022385 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:47.022345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9kn\" (UniqueName: \"kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn\") pod \"network-check-target-bthfh\" (UID: \"9593d56b-b401-4c0b-80cc-21a9e034b75d\") " pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:47.022583 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:47.022535 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:47.022583 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:47.022561 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:47.022692 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:47.022597 2569 projected.go:196] Error preparing data for projected volume kube-api-access-br9kn for pod openshift-network-diagnostics/network-check-target-bthfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:47.022692 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:47.022651 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn podName:9593d56b-b401-4c0b-80cc-21a9e034b75d nodeName:}" failed. No retries permitted until 2026-11-03 00:04:55.022633848 +0000 UTC m=+17.463297867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-br9kn" (UniqueName: "kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn") pod "network-check-target-bthfh" (UID: "9593d56b-b401-4c0b-80cc-21a9e034b75d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:47.066584 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:47.066544 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:47.066773 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:47.066693 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:04:47.066877 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:47.066773 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:47.066877 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:47.066861 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:04:47.066982 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:47.066916 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:47.067033 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:47.066985 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:04:47.163902 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:47.163858 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nk4xm" event={"ID":"fa475895-c30a-4076-98c0-d941610c151f","Type":"ContainerStarted","Data":"7457446d6ef98e25fbb60f576c611f3dbf52f7820b4ec4db39d2d7ff98e9e103"} Nov 03 00:04:48.169086 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:48.169051 2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:04:48.169749 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:48.169721 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" event={"ID":"8bc26ec7-d383-4580-a5c4-becaf9dc3278","Type":"ContainerStarted","Data":"8213425ec590d0d071343e5ea10e4de337d57e81adab7638102795389ba29cf5"} Nov 03 00:04:49.066339 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:49.066274 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:49.066540 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:49.066436 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:04:49.066847 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:49.066826 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:49.066952 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:49.066919 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:04:49.067007 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:49.066983 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:49.067069 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:49.067049 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:04:49.558602 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:49.558581 2569 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 00:04:50.022643 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:50.022509 2569 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:04:49.558599583Z","UUID":"eb48228e-a45a-4222-95c4-52f497a89dde","Handler":null,"Name":"","Endpoint":""} Nov 03 00:04:50.024110 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:50.024090 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 00:04:50.024110 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:50.024115 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 00:04:50.175272 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:50.175198 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mmcgw" event={"ID":"b3fd5ced-e9c1-4b9a-9939-473318ebbd93","Type":"ContainerStarted","Data":"26ef523745c1420a09ae0a6e7e8e6288cd18d78f8c63fdf581268316558fb282"} Nov 03 00:04:50.177996 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:50.177955 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" event={"ID":"8bc26ec7-d383-4580-a5c4-becaf9dc3278","Type":"ContainerStarted","Data":"d8621ea2f5eb12443f11d38da0399f69ccd4077ae19efe3102c6f7a5076fbc25"} Nov 03 00:04:50.179846 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:50.179790 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lk69m" event={"ID":"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b","Type":"ContainerStarted","Data":"d7b2d0e56c1b1bf9f569f81921edeb15d1f769196d81a3cfe0f0a87a17bbb513"} Nov 03 00:04:50.289181 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:50.289134 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-mmcgw" podStartSLOduration=3.484943613 podStartE2EDuration="12.289120617s" totalImagesPullingTime="8.804177004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:50.288922678 +0000 UTC m=+12.729586706" watchObservedRunningTime="2026-11-03 00:04:50.289120617 +0000 UTC m=+12.729784666" Nov 03 00:04:50.289564 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:50.289432 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nk4xm" podStartSLOduration=6.163827084 podStartE2EDuration="12.289426042s" totalImagesPullingTime="6.125598958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:47.288937018 +0000 UTC m=+9.729601050" watchObservedRunningTime="2026-11-03 00:04:50.289426042 +0000 UTC m=+12.730090152" Nov 03 00:04:50.400412 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:50.399985 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-lk69m" podStartSLOduration=3.42813812 podStartE2EDuration="12.399963578s" totalImagesPullingTime="8.971825458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:50.399634869 +0000 UTC m=+12.840298893" watchObservedRunningTime="2026-11-03 00:04:50.399963578 +0000 UTC m=+12.840627623" Nov 03 00:04:51.067199 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:51.066389 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:51.067199 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:51.066519 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:04:51.067199 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:51.066901 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:51.067199 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:51.067017 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:04:51.067199 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:51.067080 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:51.067199 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:51.067142 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:04:51.183341 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:51.183307 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" event={"ID":"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0","Type":"ContainerStarted","Data":"9374eba995344fc546ed82110f801995a97c2d7ecfdf4c3c8aeff42e87f91c04"} Nov 03 00:04:51.187972 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:51.187942 2569 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="3ee59459bf454a78999e57da19c0ef51523253f9506a8bfb62def0e521cb313c" exitCode=0 Nov 03 00:04:51.188098 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:51.188024 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7kh" event={"ID":"47d5d066-c799-4be0-9a2c-680351cb8095","Type":"ContainerDied","Data":"3ee59459bf454a78999e57da19c0ef51523253f9506a8bfb62def0e521cb313c"} Nov 03 00:04:51.190594 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:51.190559 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" event={"ID":"8bc26ec7-d383-4580-a5c4-becaf9dc3278","Type":"ContainerStarted","Data":"9470bcc85fdfeb6c48545fa82a9bda33251fe1e46014029f555439f70e72c6dc"} Nov 03 00:04:51.297213 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:51.297154 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" podStartSLOduration=1.000708643 podStartE2EDuration="12.297141332s" totalImagesPullingTime="11.296432689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:51.296673315 +0000 UTC m=+13.737337340" watchObservedRunningTime="2026-11-03 00:04:51.297141332 +0000 UTC m=+13.737805358" Nov 03 00:04:51.526654 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:51.526604 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" podStartSLOduration=3.439726778 podStartE2EDuration="13.526591577s" totalImagesPullingTime="10.086864799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:51.526447897 +0000 UTC m=+13.967111923" watchObservedRunningTime="2026-11-03 00:04:51.526591577 +0000 UTC m=+13.967255602" Nov 03 00:04:52.194744 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:52.194707 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rc4t7" event={"ID":"e744436f-8167-45f8-8e07-508d1cc24f27","Type":"ContainerStarted","Data":"3aa32842640ff45233b36470d453b28571c98d5e0c5f0db889a6be389b6f3773"} Nov 03 00:04:52.306001 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:52.305943 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rc4t7" podStartSLOduration=4.9497876 podStartE2EDuration="14.30592955s" totalImagesPullingTime="9.35614195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:52.305842079 +0000 UTC m=+14.746506104" watchObservedRunningTime="2026-11-03 00:04:52.30592955 +0000 UTC m=+14.746593576" Nov 03 00:04:53.066173 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:53.066139 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:53.066347 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:53.066197 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:53.066347 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:53.066299 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:53.066347 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:53.066298 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:04:53.066469 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:53.066376 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:04:53.066469 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:53.066448 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:04:53.199274 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:53.199238 2569 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="decc529752103d346c0f22b3543778a8f272db748776c02db84bde47f7ca49cb" exitCode=0 Nov 03 00:04:53.199852 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:53.199318 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7kh" event={"ID":"47d5d066-c799-4be0-9a2c-680351cb8095","Type":"ContainerDied","Data":"decc529752103d346c0f22b3543778a8f272db748776c02db84bde47f7ca49cb"} Nov 03 00:04:54.982186 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:54.981977 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:54.982605 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:54.982225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:54.982605 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:54.982150 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:54.982605 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:54.982326 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:54.982605 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:54.982332 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs podName:825e190f-db64-4823-8b20-4b605ba2c336 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:10.982313405 +0000 UTC m=+33.422977408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs") pod "network-metrics-daemon-zvfng" (UID: "825e190f-db64-4823-8b20-4b605ba2c336") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:54.982605 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:54.982362 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret podName:ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad nodeName:}" failed. No retries permitted until 2026-11-03 00:05:10.982352486 +0000 UTC m=+33.423016504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret") pod "global-pull-secret-syncer-dz9d5" (UID: "ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:55.066442 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:55.066400 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:55.066442 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:55.066427 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:55.066442 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:55.066448 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:55.066653 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:55.066520 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:04:55.066653 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:55.066606 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:04:55.066736 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:55.066664 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:04:55.082517 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:55.082490 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9kn\" (UniqueName: \"kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn\") pod \"network-check-target-bthfh\" (UID: \"9593d56b-b401-4c0b-80cc-21a9e034b75d\") " pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:55.082665 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:55.082654 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:55.082702 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:55.082671 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:55.082702 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:55.082680 2569 projected.go:196] Error preparing data for projected volume kube-api-access-br9kn for pod openshift-network-diagnostics/network-check-target-bthfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:55.082764 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:55.082725 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn podName:9593d56b-b401-4c0b-80cc-21a9e034b75d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.082713769 +0000 UTC m=+33.523377773 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-br9kn" (UniqueName: "kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn") pod "network-check-target-bthfh" (UID: "9593d56b-b401-4c0b-80cc-21a9e034b75d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:55.206615 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:55.206582 2569 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="6dd432a8db1b1aca436d3e938be0d2075dd1c5485b02e3d47d7270287a3894a9" exitCode=0 Nov 03 00:04:55.206810 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:55.206646 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7kh" event={"ID":"47d5d066-c799-4be0-9a2c-680351cb8095","Type":"ContainerDied","Data":"6dd432a8db1b1aca436d3e938be0d2075dd1c5485b02e3d47d7270287a3894a9"} Nov 03 00:04:56.212440 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:56.212150 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" event={"ID":"ed2418f7-c460-4057-bbc1-9d5f01278635","Type":"ContainerStarted","Data":"fe32a81e695c9faa08c90f21ccd995f3eb3dcc3b698705d1292768b5773c9cfb"} Nov 03 00:04:56.212884 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:56.212458 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" event={"ID":"ed2418f7-c460-4057-bbc1-9d5f01278635","Type":"ContainerStarted","Data":"415b074e2d00bf27dfa510ad1bf8cc471d8f7f3a02bf150da6db0de2d0f62a11"} Nov 03 00:04:56.212884 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:56.212507 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" event={"ID":"ed2418f7-c460-4057-bbc1-9d5f01278635","Type":"ContainerStarted","Data":"d858d25357c04ae9ecad0159d113178e6287139250ec336290153665df7850b4"} Nov 03 00:04:56.212884 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:56.212554 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" event={"ID":"ed2418f7-c460-4057-bbc1-9d5f01278635","Type":"ContainerStarted","Data":"04900ad157d535606a0c86dcec62190947dfbd1509786e9ed5abc5292c81c7b5"} Nov 03 00:04:56.212884 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:56.212599 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" event={"ID":"ed2418f7-c460-4057-bbc1-9d5f01278635","Type":"ContainerStarted","Data":"5b6ecd21523fddb77618684a814bccbd394f95ff7a07e569804b68c8bcef8408"} Nov 03 00:04:56.212884 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:56.212642 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" event={"ID":"ed2418f7-c460-4057-bbc1-9d5f01278635","Type":"ContainerStarted","Data":"92bed5205307b187776c18affcf199e5a707281bed10b072c2533e3cf2fb3665"} Nov 03 00:04:57.066456 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:57.066425 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:57.066676 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:57.066516 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:57.066676 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:57.066536 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:57.066676 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:57.066634 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:04:57.066857 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:57.066760 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:04:57.066928 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:57.066903 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:04:58.219276 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:58.219191 2569 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="1e47a9398ec658f9024c60bf40bba5435137f5319ea0ed96cc734de9a7ef1b53" exitCode=0 Nov 03 00:04:58.219894 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:58.219267 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7kh" event={"ID":"47d5d066-c799-4be0-9a2c-680351cb8095","Type":"ContainerDied","Data":"1e47a9398ec658f9024c60bf40bba5435137f5319ea0ed96cc734de9a7ef1b53"} Nov 03 00:04:58.222469 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:58.222447 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" event={"ID":"ed2418f7-c460-4057-bbc1-9d5f01278635","Type":"ContainerStarted","Data":"6767203dd1abc57b8d20c6bd3c8756fc65fcb815b2bf2e948288f538d1f1d869"} Nov 03 00:04:59.065815 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:59.065759 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:04:59.065815 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:59.065795 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:04:59.065815 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:59.065795 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:04:59.066045 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:59.065877 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:04:59.066045 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:59.066001 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:04:59.066110 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:04:59.066076 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:04:59.227441 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:59.227408 2569 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="d751717a3b457264054f636f89a6f9ffc9f1cbf321c175bf76423f007d44561a" exitCode=0 Nov 03 00:04:59.227885 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:04:59.227460 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7kh" event={"ID":"47d5d066-c799-4be0-9a2c-680351cb8095","Type":"ContainerDied","Data":"d751717a3b457264054f636f89a6f9ffc9f1cbf321c175bf76423f007d44561a"} Nov 03 00:05:00.233926 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:00.233635 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7kh" event={"ID":"47d5d066-c799-4be0-9a2c-680351cb8095","Type":"ContainerStarted","Data":"608994d7077974590ca530aef78a49d1d87ece1256074bf0f2919046097d9933"} Nov 03 00:05:00.236783 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:00.236743 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" event={"ID":"ed2418f7-c460-4057-bbc1-9d5f01278635","Type":"ContainerStarted","Data":"90362234ab1f962a40aece3332b819deecc4fa9b5b40df683290833348f89fdb"} Nov 03 00:05:00.237039 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:00.237021 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:00.237116 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:00.237074 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:00.252477 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:00.252453 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:00.379523 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:00.379495 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:05:00.379740 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:00.379601 2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:05:00.379992 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:00.379979 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:05:00.400012 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:00.399970 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kx7kh" podStartSLOduration=9.54860675 podStartE2EDuration="22.399958991s" totalImagesPullingTime="12.851352241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:00.399221432 +0000 UTC m=+22.839885456" watchObservedRunningTime="2026-11-03 00:05:00.399958991 +0000 UTC m=+22.840623016" Nov 03 00:05:00.683083 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:00.682999 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" podStartSLOduration=7.7591685649999995 podStartE2EDuration="22.682987643s" totalImagesPullingTime="14.923819078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:00.682841158 +0000 UTC m=+23.123505182" watchObservedRunningTime="2026-11-03 00:05:00.682987643 +0000 UTC m=+23.123651667" Nov 03 00:05:01.066479 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:01.066404 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:01.066479 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:01.066472 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:01.066684 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:01.066569 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:01.066684 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:01.066567 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:05:01.066684 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:01.066628 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:05:01.066769 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:01.066710 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:05:01.240250 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:01.240222 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:01.258642 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:01.258614 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:02.810491 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:02.810411 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bthfh"] Nov 03 00:05:02.810952 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:02.810508 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:02.810952 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:02.810619 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:05:02.813648 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:02.813620 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dz9d5"] Nov 03 00:05:02.813791 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:02.813747 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:02.813909 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:02.813881 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:05:02.860056 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:02.860033 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zvfng"] Nov 03 00:05:02.860180 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:02.860124 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:02.860217 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:02.860206 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:05:04.065889 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:04.065862 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:04.066279 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:04.065980 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:05:05.066264 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:05.066230 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:05.066264 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:05.066259 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:05.066674 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:05.066337 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:05:05.066674 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:05.066399 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:05:06.066039 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:06.066014 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:06.066192 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:06.066124 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:05:07.065906 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:07.065879 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:07.066291 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:07.065879 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:07.066291 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:07.065983 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bthfh" podUID="9593d56b-b401-4c0b-80cc-21a9e034b75d" Nov 03 00:05:07.066291 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:07.066063 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvfng" podUID="825e190f-db64-4823-8b20-4b605ba2c336" Nov 03 00:05:08.067036 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:08.066811 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:08.067365 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:08.067146 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz9d5" podUID="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" Nov 03 00:05:08.322829 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:08.322756 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeReady" Nov 03 00:05:08.322956 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:08.322880 2569 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 00:05:08.445935 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:08.445913 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lk69m_8a058bf9-45a2-44fd-908f-0e8fc41c1e5b/dns-node-resolver/0.log" Nov 03 00:05:09.066498 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:09.066464 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:09.066498 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:09.066493 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:09.118538 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:09.118511 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 00:05:09.118869 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:09.118604 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 00:05:09.168135 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:09.168111 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 00:05:09.168135 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:09.168115 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gd4nl\"" Nov 03 00:05:09.168242 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:09.168157 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fmpf5\"" Nov 03 00:05:09.481191 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:09.481171 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nk4xm_fa475895-c30a-4076-98c0-d941610c151f/node-ca/0.log" Nov 03 00:05:10.066539 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:10.066508 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:10.121284 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:10.121261 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vqr8l\"" Nov 03 00:05:10.121574 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:10.121335 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 00:05:11.003335 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:11.003298 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:11.003523 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:11.003355 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:11.003523 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:11.003449 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 00:05:11.003523 ip-10-0-140-172 kubenswrapper[2569]: E1103 00:05:11.003513 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs podName:825e190f-db64-4823-8b20-4b605ba2c336 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:43.003495721 +0000 UTC m=+65.444159738 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs") pod "network-metrics-daemon-zvfng" (UID: "825e190f-db64-4823-8b20-4b605ba2c336") : secret "metrics-daemon-secret" not found Nov 03 00:05:11.007162 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:11.007143 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:11.103821 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:11.103795 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9kn\" (UniqueName: \"kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn\") pod \"network-check-target-bthfh\" (UID: \"9593d56b-b401-4c0b-80cc-21a9e034b75d\") " pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:11.106436 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:11.106411 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br9kn\" (UniqueName: \"kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn\") pod \"network-check-target-bthfh\" (UID: \"9593d56b-b401-4c0b-80cc-21a9e034b75d\") " pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:11.181742 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:11.181713 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:11.275363 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:11.275302 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:11.555455 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:11.555377 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bthfh"] Nov 03 00:05:11.558583 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:05:11.558560 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9593d56b_b401_4c0b_80cc_21a9e034b75d.slice/crio-da3ff9c56bb8ab602bb8fbd1fd505c52f26dc8e43163e10f4019e3a67c4eee33 WatchSource:0}: Error finding container da3ff9c56bb8ab602bb8fbd1fd505c52f26dc8e43163e10f4019e3a67c4eee33: Status 404 returned error can't find the container with id da3ff9c56bb8ab602bb8fbd1fd505c52f26dc8e43163e10f4019e3a67c4eee33 Nov 03 00:05:11.663502 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:11.663468 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dz9d5"] Nov 03 00:05:11.666899 ip-10-0-140-172 kubenswrapper[2569]: W1103 00:05:11.666873 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5a64df_b2ce_4bd0_b4eb_97144bcb9aad.slice/crio-8f364efebd865a5f1afc28709682d112deadc0be71a6c237cba846c8f91695aa WatchSource:0}: Error finding container 8f364efebd865a5f1afc28709682d112deadc0be71a6c237cba846c8f91695aa: Status 404 returned error can't find the container with id 8f364efebd865a5f1afc28709682d112deadc0be71a6c237cba846c8f91695aa Nov 03 00:05:12.265660 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:12.265618 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dz9d5" event={"ID":"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad","Type":"ContainerStarted","Data":"8f364efebd865a5f1afc28709682d112deadc0be71a6c237cba846c8f91695aa"} Nov 03 00:05:12.266731 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:12.266701 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bthfh" event={"ID":"9593d56b-b401-4c0b-80cc-21a9e034b75d","Type":"ContainerStarted","Data":"da3ff9c56bb8ab602bb8fbd1fd505c52f26dc8e43163e10f4019e3a67c4eee33"} Nov 03 00:05:16.278574 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:16.278506 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bthfh" event={"ID":"9593d56b-b401-4c0b-80cc-21a9e034b75d","Type":"ContainerStarted","Data":"ae9eb62336be02e73d68886ca9c3b558d60cd9365edce21531ca67649f65adcd"} Nov 03 00:05:16.279055 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:16.278652 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:16.390732 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:16.390658 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bthfh" podStartSLOduration=34.710417268 podStartE2EDuration="38.390639053s" totalImagesPullingTime="3.680221785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:16.390264766 +0000 UTC m=+38.830928792" watchObservedRunningTime="2026-11-03 00:05:16.390639053 +0000 UTC m=+38.831303079" Nov 03 00:05:16.668320 ip-10-0-140-172 kubenswrapper[2569]: I1103 00:05:16.668209 2569 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:05:16.668349 ip-10-0-140-172 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 00:05:16.678204 ip-10-0-140-172 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 00:05:16.678364 ip-10-0-140-172 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 00:05:16.678410 ip-10-0-140-172 systemd[1]: kubelet.service: Consumed 2.111s CPU time. Nov 03 00:05:16.687471 ip-10-0-140-172 systemd[1]: Starting Kubernetes Kubelet... Nov 03 00:05:16.746230 ip-10-0-140-172 kubenswrapper[4304]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:05:16.746230 ip-10-0-140-172 kubenswrapper[4304]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 00:05:16.746230 ip-10-0-140-172 kubenswrapper[4304]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:05:16.746230 ip-10-0-140-172 kubenswrapper[4304]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:05:16.748207 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748139 4304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:16.748207 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748208 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748212 4304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748215 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748217 4304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748220 4304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748222 4304 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748225 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748227 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748230 4304 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748232 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748235 4304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748237 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748240 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748242 4304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748245 4304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748247 4304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748250 4304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748252 4304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748255 4304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:16.748273 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748258 4304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748261 4304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748263 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748266 4304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748269 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748271 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748286 4304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748289 4304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748291 4304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748293 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748296 4304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748298 4304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748301 4304 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748303 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748306 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748308 4304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748311 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748313 4304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748315 4304 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748317 4304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:16.748724 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748320 4304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748322 4304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748324 4304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748327 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748329 4304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748332 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748335 4304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748338 4304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748340 4304 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748343 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748345 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748349 4304 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748352 4304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748355 4304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748357 4304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748360 4304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748362 4304 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748365 4304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748367 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:16.749258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748370 4304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748373 4304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748375 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748377 4304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748380 4304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748383 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748385 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748387 4304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748389 4304 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748392 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748394 4304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748397 4304 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748399 4304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748402 4304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748404 4304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748406 4304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748409 4304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748411 4304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748414 4304 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748417 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:16.749726 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748419 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748422 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748426 4304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748429 4304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748433 4304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748436 4304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748439 4304 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748442 4304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748444 4304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748446 4304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748448 4304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748452 4304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748455 4304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748457 4304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748460 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748462 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748465 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748467 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748469 4304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:16.750243 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748472 4304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748474 4304 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748477 4304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748479 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748481 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748484 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748486 4304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.748489 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749289 4304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749296 4304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749300 4304 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749302 4304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749309 4304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749311 4304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749314 4304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749317 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749319 4304 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749322 4304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749324 4304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749327 4304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:16.750701 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749329 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749332 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749334 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749336 4304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749339 4304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749341 4304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749344 4304 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749347 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749349 4304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749352 4304 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749354 4304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749357 4304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749359 4304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749362 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749364 4304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749367 4304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749369 4304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749372 4304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749375 4304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749377 4304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:16.751201 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749380 4304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749382 4304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749384 4304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749387 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749390 4304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749393 4304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749395 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749398 4304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749400 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749403 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749405 4304 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749408 4304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749410 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749413 4304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749415 4304 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749418 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749420 4304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749423 4304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749425 4304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749428 4304 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:16.751676 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749431 4304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749433 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749436 4304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749438 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749441 4304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749443 4304 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749446 4304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749448 4304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749453 4304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749456 4304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749459 4304 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749462 4304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749465 4304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749467 4304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749469 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749472 4304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749475 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749478 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749481 4304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:16.752174 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749483 4304 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749485 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749488 4304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749490 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749493 4304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749495 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749498 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749500 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749502 4304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749505 4304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749507 4304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749510 4304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749512 4304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749515 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749518 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749520 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749523 4304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749525 4304 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749528 4304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:16.752655 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749530 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749532 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749535 4304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749537 4304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749540 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749543 4304 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749545 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749548 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749550 4304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749552 4304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749556 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749559 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749562 4304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749567 4304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749570 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.749572 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750054 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750059 4304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750062 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:16.753195 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750065 4304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750067 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750070 4304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750073 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750075 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750078 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750080 4304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750082 4304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750085 4304 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750088 4304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750090 4304 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750093 4304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750096 4304 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750098 4304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750102 4304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750105 4304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750108 4304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750111 4304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750113 4304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:16.753673 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750115 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750118 4304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750120 4304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750126 4304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750130 4304 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750133 4304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750136 4304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750138 4304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750140 4304 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750143 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750145 4304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750148 4304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750150 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750153 4304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750155 4304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750158 4304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750160 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750162 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750165 4304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:16.754186 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750167 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750170 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750172 4304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750175 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750177 4304 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750180 4304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750183 4304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750186 4304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750188 4304 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750191 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750193 4304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750196 4304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750198 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750200 4304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750203 4304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750205 4304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750209 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750213 4304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750216 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750218 4304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:16.754635 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750221 4304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750223 4304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750226 4304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750228 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750231 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750233 4304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750235 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750238 4304 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750241 4304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750243 4304 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750246 4304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750248 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750251 4304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750254 4304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750256 4304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750259 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750262 4304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750264 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750267 4304 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750269 4304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:16.755137 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750271 4304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750274 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750276 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750279 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750281 4304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750283 4304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750286 4304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750288 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750292 4304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750296 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750298 4304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750301 4304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750303 4304 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750306 4304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750308 4304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750311 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750313 4304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750316 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750318 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:16.755615 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750320 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750323 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750325 4304 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750328 4304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750331 4304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.750333 4304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750433 4304 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750444 4304 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750454 4304 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750461 4304 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750466 4304 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750469 4304 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750474 4304 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750477 4304 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750480 4304 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750483 4304 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750486 4304 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750489 4304 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750492 4304 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750495 4304 flags.go:64] FLAG: --cgroup-root="" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750498 4304 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750503 4304 flags.go:64] FLAG: --client-ca-file="" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750507 4304 flags.go:64] FLAG: --cloud-provider="external" Nov 03 00:05:16.756075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750509 4304 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750513 4304 flags.go:64] FLAG: --cluster-domain="" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750516 4304 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750519 4304 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750522 4304 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750525 4304 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750529 4304 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750532 4304 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750535 4304 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750538 4304 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750540 4304 flags.go:64] FLAG: --contention-profiling="false" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750543 4304 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750546 4304 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750549 4304 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750552 4304 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750556 4304 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750559 4304 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750561 4304 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750564 4304 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750567 4304 flags.go:64] FLAG: --enable-server="true" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750570 4304 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750574 4304 flags.go:64] FLAG: --event-burst="100" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750577 4304 flags.go:64] FLAG: --event-qps="50" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750581 4304 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 00:05:16.756609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750583 4304 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750586 4304 flags.go:64] FLAG: --eviction-hard="" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750590 4304 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750593 4304 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750596 4304 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750598 4304 flags.go:64] FLAG: --eviction-soft="" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750603 4304 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750607 4304 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750610 4304 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750613 4304 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750615 4304 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750618 4304 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750621 4304 flags.go:64] FLAG: --feature-gates="" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750625 4304 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750628 4304 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750631 4304 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750634 4304 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750637 4304 flags.go:64] FLAG: --healthz-port="10248" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750640 4304 flags.go:64] FLAG: --help="false" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750642 4304 flags.go:64] FLAG: --hostname-override="ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750645 4304 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750648 4304 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750651 4304 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 00:05:16.757482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750654 4304 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750657 4304 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750660 4304 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750663 4304 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750665 4304 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750668 4304 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750672 4304 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750675 4304 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750677 4304 flags.go:64] FLAG: --kube-reserved="" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750680 4304 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750683 4304 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750686 4304 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750689 4304 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750691 4304 flags.go:64] FLAG: --lock-file="" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750694 4304 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750697 4304 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750702 4304 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750707 4304 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750710 4304 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750713 4304 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750716 4304 flags.go:64] FLAG: --logging-format="text" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750719 4304 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750722 4304 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750725 4304 flags.go:64] FLAG: --manifest-url="" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750728 4304 flags.go:64] FLAG: --manifest-url-header="" Nov 03 00:05:16.758447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750732 4304 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750735 4304 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750739 4304 flags.go:64] FLAG: --max-pods="110" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750742 4304 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750744 4304 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750747 4304 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750750 4304 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750753 4304 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750756 4304 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750759 4304 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750765 4304 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750768 4304 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750784 4304 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750787 4304 flags.go:64] FLAG: --pod-cidr="" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750790 4304 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750793 4304 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750796 4304 flags.go:64] FLAG: --pods-per-core="0" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750799 4304 flags.go:64] FLAG: --port="10250" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750802 4304 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750805 4304 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a9fcc8855f4d508a" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750808 4304 flags.go:64] FLAG: --qos-reserved="" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750811 4304 flags.go:64] FLAG: --read-only-port="10255" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750816 4304 flags.go:64] FLAG: --register-node="true" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750819 4304 flags.go:64] FLAG: --register-with-taints="" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750823 4304 flags.go:64] FLAG: --registry-burst="10" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750826 4304 flags.go:64] FLAG: --registry-qps="5" Nov 03 00:05:16.759500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750828 4304 flags.go:64] FLAG: --reserved-cpus="" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750831 4304 flags.go:64] FLAG: --reserved-memory="" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750835 4304 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750838 4304 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750840 4304 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750843 4304 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750846 4304 flags.go:64] FLAG: --runonce="false" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750849 4304 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750852 4304 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750855 4304 flags.go:64] FLAG: --seccomp-default="false" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750858 4304 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750860 4304 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750863 4304 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750866 4304 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750869 4304 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750871 4304 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750874 4304 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750877 4304 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750880 4304 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750882 4304 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750885 4304 flags.go:64] FLAG: --system-cgroups="" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750888 4304 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750893 4304 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750896 4304 flags.go:64] FLAG: --tls-cert-file="" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750899 4304 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 00:05:16.760572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750902 4304 flags.go:64] FLAG: --tls-min-version="" Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750905 4304 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750908 4304 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750911 4304 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750915 4304 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750917 4304 flags.go:64] FLAG: --v="2" Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750922 4304 flags.go:64] FLAG: --version="false" Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750926 4304 flags.go:64] FLAG: --vmodule="" Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750929 4304 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.750932 4304 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751054 4304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751057 4304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751059 4304 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751062 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751064 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751067 4304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751069 4304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751071 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751074 4304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751077 4304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751079 4304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:16.761789 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751081 4304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751084 4304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751086 4304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751089 4304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751091 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751094 4304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751096 4304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751099 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751101 4304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751104 4304 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751106 4304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751109 4304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751111 4304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751114 4304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751118 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751120 4304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751123 4304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751125 4304 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751128 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751130 4304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:16.762349 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751133 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751135 4304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751138 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751140 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751142 4304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751145 4304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751147 4304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751150 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751152 4304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751154 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751157 4304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751159 4304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751162 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751164 4304 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751166 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751169 4304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751171 4304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751173 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751176 4304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:16.762934 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751178 4304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751180 4304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751183 4304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751186 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751188 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751190 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751193 4304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751196 4304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751199 4304 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751201 4304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751203 4304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751206 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751208 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751211 4304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751213 4304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751216 4304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751219 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751221 4304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751223 4304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:16.763578 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751226 4304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751228 4304 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751232 4304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751236 4304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751239 4304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751243 4304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751246 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751248 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751250 4304 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751253 4304 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751255 4304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751258 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751260 4304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751262 4304 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751265 4304 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751267 4304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751270 4304 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751273 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751275 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751278 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:16.764119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751281 4304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751284 4304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751288 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751290 4304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751293 4304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751295 4304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751298 4304 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751300 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751303 4304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751305 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751307 4304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751310 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751312 4304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751315 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751317 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751321 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:16.764636 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751324 4304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.751348 4304 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751381 4304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751385 4304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751388 4304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751391 4304 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751393 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751396 4304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751398 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751400 4304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751403 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751405 4304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751408 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751411 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751413 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751416 4304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:16.765847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751419 4304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751421 4304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751423 4304 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751426 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751428 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751430 4304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751433 4304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751435 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751437 4304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751440 4304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751442 4304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751445 4304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751447 4304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751449 4304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751452 4304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751455 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751458 4304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751460 4304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751462 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:16.766258 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751465 4304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751467 4304 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751469 4304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751472 4304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751474 4304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751477 4304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751479 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751481 4304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751484 4304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751486 4304 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751489 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751491 4304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751494 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751496 4304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751499 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751501 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751504 4304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751506 4304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751508 4304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751511 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:16.766707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751513 4304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751515 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751518 4304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751520 4304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751523 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751525 4304 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751527 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751530 4304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751533 4304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751536 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751539 4304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751541 4304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751544 4304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751546 4304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751548 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751551 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751553 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751556 4304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751558 4304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:16.767231 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751560 4304 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751562 4304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751565 4304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751567 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751569 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751572 4304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751574 4304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751577 4304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751579 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751582 4304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751584 4304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751586 4304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751588 4304 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751592 4304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751595 4304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751598 4304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751601 4304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751603 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751605 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:16.767674 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751608 4304 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751610 4304 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751614 4304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751617 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751619 4304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751622 4304 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751625 4304 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751627 4304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751630 4304 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751633 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751635 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751637 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751640 4304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751644 4304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.751648 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.755550 4304 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.755564 4304 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.755581 4304 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.755585 4304 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 00:05:16.768134 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755651 4304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755656 4304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755660 4304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755663 4304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755666 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755669 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755672 4304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755675 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755677 4304 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755680 4304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755683 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755685 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755688 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755690 4304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755692 4304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755695 4304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755697 4304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755700 4304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755702 4304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755705 4304 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:16.768616 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755707 4304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755710 4304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755712 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755714 4304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755717 4304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755719 4304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755722 4304 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755724 4304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755727 4304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755730 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755733 4304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755735 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755738 4304 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755740 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755743 4304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755745 4304 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755748 4304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755750 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755753 4304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:16.769136 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755755 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755757 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755760 4304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755762 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755765 4304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755767 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755770 4304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755805 4304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755808 4304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755811 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755813 4304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755816 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755818 4304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755820 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755823 4304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755825 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755828 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755831 4304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755833 4304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755836 4304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:16.769597 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755838 4304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755840 4304 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755844 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755846 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755849 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755851 4304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755853 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755856 4304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755858 4304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755861 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755863 4304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755865 4304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755867 4304 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755870 4304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755872 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755874 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755877 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755880 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755883 4304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755888 4304 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:16.770112 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755891 4304 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755894 4304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755896 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755898 4304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755901 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755903 4304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755906 4304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755908 4304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755910 4304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755913 4304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755915 4304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755918 4304 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755920 4304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755922 4304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755925 4304 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755927 4304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755930 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755932 4304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755935 4304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:16.770599 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755937 4304 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755939 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755942 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755944 4304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755948 4304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755952 4304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755955 4304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.755958 4304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.755985 4304 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756108 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756113 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756116 4304 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756119 4304 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756121 4304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756124 4304 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756126 4304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:16.771069 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756129 4304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756131 4304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756133 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756136 4304 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756138 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756141 4304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756143 4304 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756145 4304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756148 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756150 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756153 4304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756155 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756157 4304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756160 4304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756163 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756165 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756168 4304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756170 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756172 4304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756174 4304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:16.771451 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756177 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756179 4304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756182 4304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756184 4304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756187 4304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756189 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756191 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756194 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756196 4304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756200 4304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756203 4304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756206 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756208 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756211 4304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756213 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756215 4304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756218 4304 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756220 4304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756222 4304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756225 4304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:16.771926 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756227 4304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756230 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756232 4304 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756234 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756236 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756239 4304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756241 4304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756244 4304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756246 4304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756248 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756251 4304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756253 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756255 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756257 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756260 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756262 4304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756265 4304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756267 4304 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756269 4304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:16.772394 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756272 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756274 4304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756277 4304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756279 4304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756282 4304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756284 4304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756287 4304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756291 4304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756294 4304 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756296 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756299 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756301 4304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756303 4304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756306 4304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756308 4304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756310 4304 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756313 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756315 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756317 4304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:16.772885 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756320 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756322 4304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756324 4304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756327 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756333 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756336 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756338 4304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756341 4304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756343 4304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756345 4304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756348 4304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756351 4304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756353 4304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756355 4304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756362 4304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756365 4304 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756368 4304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756370 4304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756373 4304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756375 4304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:16.773366 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756377 4304 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.756399 4304 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756426 4304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756430 4304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756432 4304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756435 4304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756438 4304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756440 4304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756443 4304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756445 4304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756447 4304 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756450 4304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756452 4304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756454 4304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756457 4304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756459 4304 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:16.774006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756462 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756466 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756468 4304 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756471 4304 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756473 4304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756475 4304 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756478 4304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756480 4304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756483 4304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756485 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756489 4304 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756491 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756494 4304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756496 4304 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756499 4304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756501 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756504 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756506 4304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756509 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756511 4304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:16.774632 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756514 4304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756516 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756519 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756521 4304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756524 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756526 4304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756528 4304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756531 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756533 4304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756535 4304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756538 4304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756540 4304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756543 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756547 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756549 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756552 4304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756556 4304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756560 4304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756563 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:16.775216 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756565 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756568 4304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756570 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756574 4304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756577 4304 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756579 4304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756582 4304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756585 4304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756587 4304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756589 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756592 4304 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756594 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756596 4304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756599 4304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756601 4304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756603 4304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756606 4304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756608 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756611 4304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:16.775710 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756613 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756616 4304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756618 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756621 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756623 4304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756625 4304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756628 4304 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756632 4304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756635 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756637 4304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756640 4304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756642 4304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756644 4304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756647 4304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756650 4304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756654 4304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756659 4304 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756662 4304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756664 4304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:16.776182 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756667 4304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756669 4304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756672 4304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756674 4304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756677 4304 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756679 4304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756681 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756684 4304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756686 4304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756689 4304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756691 4304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756693 4304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756696 4304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756699 4304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:16.756701 4304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.756968 4304 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.759052 4304 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.759118 4304 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 00:05:16.776633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.759398 4304 server.go:1014] "Starting client certificate rotation" Nov 03 00:05:16.777183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.759536 4304 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:05:16.777183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.759588 4304 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:38 +0000 UTC" deadline="2028-05-24 19:46:02.395776868 +0000 UTC" Nov 03 00:05:16.777183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.759614 4304 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13651h40m45.636164416s" Nov 03 00:05:16.777183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.760115 4304 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:05:16.777183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.761812 4304 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:05:16.777183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.763210 4304 log.go:25] "Validated CRI v1 runtime API" Nov 03 00:05:16.777183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.764645 4304 log.go:25] "Validated CRI v1 image API" Nov 03 00:05:16.777183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.765430 4304 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 00:05:16.777183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.766694 4304 fs.go:135] Filesystem UUIDs: map[48a685b5-c35f-4506-a580-b8c85d6f1746:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d176709f-55ad-46a9-bce1-bfaa99db4f13:/dev/nvme0n1p3] Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.766712 4304 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18f04304a2355eb8ec06ee6f09511152a167e79e98b977ac62db21d88e394809/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18f04304a2355eb8ec06ee6f09511152a167e79e98b977ac62db21d88e394809/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40fafab6756a07481e1c69da53921402df85d8479a7222b64282dcce0e456272/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40fafab6756a07481e1c69da53921402df85d8479a7222b64282dcce0e456272/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46198f6a87e25b1adf1568ca64c975901243d2a1d33d4a81d8ed313568007dd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46198f6a87e25b1adf1568ca64c975901243d2a1d33d4a81d8ed313568007dd1/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d476c7e47d2c99c05df779167116f85636f440efdc4f4ac5bcb5fb20c3f7c39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d476c7e47d2c99c05df779167116f85636f440efdc4f4ac5bcb5fb20c3f7c39/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f364efebd865a5f1afc28709682d112deadc0be71a6c237cba846c8f91695aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f364efebd865a5f1afc28709682d112deadc0be71a6c237cba846c8f91695aa/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/add431a4b4a0904c81c9808cc3b4724f9759184f64b2854c4bebb7f78e1f33e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/add431a4b4a0904c81c9808cc3b4724f9759184f64b2854c4bebb7f78e1f33e5/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b422b0aac24babb59aa37e41db436ac16ae2c9076b0c6986f70fbf2a15cb3bd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b422b0aac24babb59aa37e41db436ac16ae2c9076b0c6986f70fbf2a15cb3bd3/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4533793b83e46b1869964c16b94011bc06c00ea9a0d05afd66cdd4b17eeb784/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4533793b83e46b1869964c16b94011bc06c00ea9a0d05afd66cdd4b17eeb784/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfcd096023b1bf068a242ef678870dd8f889f5cdfeb17ed1944ce715a52c1f1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfcd096023b1bf068a242ef678870dd8f889f5cdfeb17ed1944ce715a52c1f1f/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d19b52601e459e2ea430b7723b2510275d128ca9d96c89f9a6003efa6a937288/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d19b52601e459e2ea430b7723b2510275d128ca9d96c89f9a6003efa6a937288/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d39c93f03caa50f35b40e92aedf5376c8cf5911d789f150e8cfaa2e77af45bd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d39c93f03caa50f35b40e92aedf5376c8cf5911d789f150e8cfaa2e77af45bd0/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da3ff9c56bb8ab602bb8fbd1fd505c52f26dc8e43163e10f4019e3a67c4eee33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da3ff9c56bb8ab602bb8fbd1fd505c52f26dc8e43163e10f4019e3a67c4eee33/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/434003bf-315d-42b3-8ee4-9cdc53936d73/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/434003bf-315d-42b3-8ee4-9cdc53936d73/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47d5d066-c799-4be0-9a2c-680351cb8095/volumes/kubernetes.io~projected/kube-api-access-lk4dj:{mountpoint:/var/lib/kubelet/pods/47d5d066-c799-4be0-9a2c-680351cb8095/volumes/kubernetes.io~projected/kube-api-access-lk4dj major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0/volumes/kubernetes.io~projected/kube-api-access-qq7js:{mountpoint:/var/lib/kubelet/pods/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0/volumes/kubernetes.io~projected/kube-api-access-qq7js major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/825e190f-db64-4823-8b20-4b605ba2c336/volumes/kubernetes.io~projected/kube-api-access-vhwlj:{mountpoint:/var/lib/kubelet/pods/825e190f-db64-4823-8b20-4b605ba2c336/volumes/kubernetes.io~projected/kube-api-access-vhwlj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b/volumes/kubernetes.io~projected/kube-api-access-4rwgs:{mountpoint:/var/lib/kubelet/pods/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b/volumes/kubernetes.io~projected/kube-api-access-4rwgs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bc26ec7-d383-4580-a5c4-becaf9dc3278/volumes/kubernetes.io~projected/kube-api-access-7kjs8:{mountpoint:/var/lib/kubelet/pods/8bc26ec7-d383-4580-a5c4-becaf9dc3278/volumes/kubernetes.io~projected/kube-api-access-7kjs8 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9593d56b-b401-4c0b-80cc-21a9e034b75d/volumes/kubernetes.io~projected/kube-api-access-br9kn:{mountpoint:/var/lib/kubelet/pods/9593d56b-b401-4c0b-80cc-21a9e034b75d/volumes/kubernetes.io~projected/kube-api-access-br9kn major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3fd5ced-e9c1-4b9a-9939-473318ebbd93/volumes/kubernetes.io~projected/kube-api-access-h4nfn:{mountpoint:/var/lib/kubelet/pods/b3fd5ced-e9c1-4b9a-9939-473318ebbd93/volumes/kubernetes.io~projected/kube-api-access-h4nfn major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e744436f-8167-45f8-8e07-508d1cc24f27/volumes/kubernetes.io~projected/kube-api-access-fnq68:{mountpoint:/var/lib/kubelet/pods/e744436f-8167-45f8-8e07-508d1cc24f27/volumes/kubernetes.io~projected/kube-api-access-fnq68 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed2418f7-c460-4057-bbc1-9d5f01278635/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ed2418f7-c460-4057-bbc1-9d5f01278635/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed2418f7-c460-4057-bbc1-9d5f01278635/volumes/kubernetes.io~projected/kube-api-access-m4ctz:{mountpoint:/var/lib/kubelet/pods/ed2418f7-c460-4057-bbc1-9d5f01278635/volumes/kubernetes.io~projected/kube-api-access-m4ctz major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed2418f7-c460-4057-bbc1-9d5f01278635/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ed2418f7-c460-4057-bbc1-9d5f01278635/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa475895-c30a-4076-98c0-d941610c151f/volumes/kubernetes.io~projected/kube-api-access-glqv6:{mountpoint:/var/lib/kubelet/pods/fa475895-c30a-4076-98c0-d941610c151f/volumes/kubernetes.io~projected/kube-api-access-glqv6 major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/072b3071b8bf853c200776d2bdf5ccdaf502432c73191a48bdb7f905c769a745/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7e74a5f25da34840c37eafc15837fd0e0b9496e40049cdcc189db7dc2602e5b6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2684748b52c0373dd6ba9677049f7ac3447d606274a00e4c31f47c6289473761/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/679d1c3629d061c4499eb3d6b8be5f6a783df212ebb394f128a2d7c83e98e14e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/56f5d3211e54acd78c953dfd7c0f5687bc4225fc365e70362c437a86dc620fbc/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e345c9cf7498c99ec4db0f9cf70bb1e5eb48ac157e8a18128c33be9d1bc2f214/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f3aee88e523e469418917069a21abbc3804c9a9636a6e4b2bacf8997e512dd36/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6266b0a15427d19aeda1a43fb0a0022f48f05c2306cd66e8f2ab87a7c295c902/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/be00c32e207c064c004f934f0c855f775e39c4706dc74d7e5374cda68f81c551/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/97588a5d2b49df080257e7ebc51dec3328278a92fc526a9fa3af57f34eeef109/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/0afb0731452fd82554b3eeae8c30831debec100fcc16274fa41a5a24c3791416/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/0bc0ba17f507da31853f01d956e33846fa110e7b4089db7e529335879e7dc97f/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/7fc1e41aae36e36494612d4c0f20568707339b9d650edd03e17ad5abdf3f832c/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cc1e56768fe77665e8c484ee949279972f82a1782afde1d3f773fbc3721971c5/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2948fd4ae76428e6b59387c6e9a4f5620d2e41344eba880cefbb519413960dca/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d257c19450132e255acaaf470a2e03c9436a49fddcf613032741c77a504ec8c3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/83951d533c0a2a9a9f79262a767e45228b1a955dbb7067ee4c6e9310a66953eb/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b4cc2e456332742ce20e9d7eefe67be213a4ad766f34e30991db0613f317f1eb/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a3e6b4c8232adc6ffec68f5567c71ac59a6e87bc6f309f8008193faf10930503/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b2464972769341e54fe6c7a6e66ee8aaa75d102d98ee0a1dab62e262fffe3b69/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/bf757078d5a1f7420e483222431e4824912b7279ccdae573a5ce8794ffc92137/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/3a26628d530e2be122a1bbae18fff98c3cdbd8b081bf0bf8c64e1d6bf65b8fbd/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/effc18fe98df68707057efc0b7d1814a2a45ce0e2ccd7e4036e6cdae706d359a/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/3fab4787a7ac31dab522ba93924eac87731070c549ff694b954745274e72fc08/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/e50bada1448dc4f4d1c19eea4f2f0cba3d5fee743018dc03f9d04315068b275c/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/08afd520992c80ac124e39f7c8efd3a1e892911229967e693d5f1a1761397c2c/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/5a69bd6c00a70dd0fe15e8c9f56da04baa13a7ce87852df1bdb5eb88ba08d540/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/87f00c169e23f7cc5a75b9daa122b7d43a7b0fc3f2cd973f18ad154ecb12655c/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/3280671df5588002523b6807cd2295c0bf496131fa9c779d1cfc0aa57b7562c5/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/fef35ac60deed00fb340c466057f030c1ab216a01e67c34509a75db48b363820/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ef4a5bb33074a2e19dfc1d62a6589cbcc0e1e4bc6bbbd73b18892bfb1efba7a7/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/3875fb696891012bcd88b271486acc8eeccb01e76692bd51a3268b0739e79ba0/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/68165edc6e7589f835595f11892d320da98f1ce5d5741d53c307d4d3c48add14/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/c872fbae927f3024c5dae256b21c84546f82b1a87de5812a937bdb83ab6024b2/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5f42a1f4dcdbdb3c4b288a216cd55112a4a77a3b1ba76a3797b51ef742435a92/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.772626 4304 manager.go:217] Machine: {Timestamp:2026-11-03 00:05:16.772183777 +0000 UTC m=+0.055741442 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108725 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2263daf80fa0f9e45e9042819d436f SystemUUID:ec2263da-f80f-a0f9-e45e-9042819d436f BootID:97d4bf86-a878-42b9-984d-23e13667ea98 Filesystems:[{Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/434003bf-315d-42b3-8ee4-9cdc53936d73/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fa475895-c30a-4076-98c0-d941610c151f/volumes/kubernetes.io~projected/kube-api-access-glqv6 DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/add431a4b4a0904c81c9808cc3b4724f9759184f64b2854c4bebb7f78e1f33e5/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d39c93f03caa50f35b40e92aedf5376c8cf5911d789f150e8cfaa2e77af45bd0/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46198f6a87e25b1adf1568ca64c975901243d2a1d33d4a81d8ed313568007dd1/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8bc26ec7-d383-4580-a5c4-becaf9dc3278/volumes/kubernetes.io~projected/kube-api-access-7kjs8 DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ed2418f7-c460-4057-bbc1-9d5f01278635/volumes/kubernetes.io~projected/kube-api-access-m4ctz DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/825e190f-db64-4823-8b20-4b605ba2c336/volumes/kubernetes.io~projected/kube-api-access-vhwlj DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40fafab6756a07481e1c69da53921402df85d8479a7222b64282dcce0e456272/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e744436f-8167-45f8-8e07-508d1cc24f27/volumes/kubernetes.io~projected/kube-api-access-fnq68 DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b/volumes/kubernetes.io~projected/kube-api-access-4rwgs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f364efebd865a5f1afc28709682d112deadc0be71a6c237cba846c8f91695aa/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed2418f7-c460-4057-bbc1-9d5f01278635/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b422b0aac24babb59aa37e41db436ac16ae2c9076b0c6986f70fbf2a15cb3bd3/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da3ff9c56bb8ab602bb8fbd1fd505c52f26dc8e43163e10f4019e3a67c4eee33/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3fd5ced-e9c1-4b9a-9939-473318ebbd93/volumes/kubernetes.io~projected/kube-api-access-h4nfn DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/47d5d066-c799-4be0-9a2c-680351cb8095/volumes/kubernetes.io~projected/kube-api-access-lk4dj DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0/volumes/kubernetes.io~projected/kube-api-access-qq7js DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d476c7e47d2c99c05df779167116f85636f440efdc4f4ac5bcb5fb20c3f7c39/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4533793b83e46b1869964c16b94011bc06c00ea9a0d05afd66cdd4b17eeb784/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d19b52601e459e2ea430b7723b2510275d128ca9d96c89f9a6003efa6a937288/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18f04304a2355eb8ec06ee6f09511152a167e79e98b977ac62db21d88e394809/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfcd096023b1bf068a242ef678870dd8f889f5cdfeb17ed1944ce715a52c1f1f/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed2418f7-c460-4057-bbc1-9d5f01278635/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9593d56b-b401-4c0b-80cc-21a9e034b75d/volumes/kubernetes.io~projected/kube-api-access-br9kn DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:8f364efebd865a5 MacAddress:2e:9a:3b:1a:a2:c6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:03:56:af:06:df Speed:0 Mtu:9001} {Name:br-int MacAddress:0e:67:ad:a5:ea:96 Speed:0 Mtu:8901} {Name:da3ff9c56bb8ab6 MacAddress:8e:60:a3:2e:1f:a0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:03:56:af:06:df Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b6:fa:a2:e6:92:30 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c6:3a:61:13:0c:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.772872 4304 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.772912 4304 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773168 4304 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773183 4304 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-172.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773300 4304 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773307 4304 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773317 4304 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773329 4304 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773399 4304 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773570 4304 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773685 4304 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773719 4304 kubelet.go:493] "Attempting to sync node with API server" Nov 03 00:05:16.777428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773735 4304 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 00:05:16.778011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773745 4304 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 00:05:16.778011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773753 4304 kubelet.go:405] "Adding apiserver pod source" Nov 03 00:05:16.778011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.773762 4304 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 00:05:16.778011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.774474 4304 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 00:05:16.778011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.774490 4304 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:05:16.778011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.775354 4304 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 00:05:16.778156 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778117 4304 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 00:05:16.778156 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778129 4304 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:05:16.778229 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778218 4304 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 00:05:16.778401 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778391 4304 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 00:05:16.778437 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778419 4304 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 00:05:16.778511 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778503 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 00:05:16.778540 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778517 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 00:05:16.778540 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778523 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 00:05:16.778540 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778528 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 00:05:16.778540 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778534 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 00:05:16.778540 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778539 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 00:05:16.778756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778545 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 00:05:16.778756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778551 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 00:05:16.778756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778558 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 00:05:16.778756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778564 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 00:05:16.778756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778571 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 00:05:16.778756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778580 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 00:05:16.778756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778601 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 00:05:16.778756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.778606 4304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 00:05:16.781669 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.781657 4304 server.go:1263] "Started kubelet" Nov 03 00:05:16.781872 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.781763 4304 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 00:05:16.782339 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.781867 4304 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 00:05:16.782339 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.782019 4304 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 00:05:16.782339 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.782285 4304 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 00:05:16.782363 ip-10-0-140-172 systemd[1]: Started Kubernetes Kubelet. Nov 03 00:05:16.783149 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.783111 4304 server.go:317] "Adding debug handlers to kubelet server" Nov 03 00:05:16.784114 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.784097 4304 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 00:05:16.784212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.784131 4304 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 00:05:16.784212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.784165 4304 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 00:05:16.784212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.784177 4304 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 00:05:16.784361 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.784201 4304 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:39 +0000 UTC" deadline="2028-04-20 11:28:27.707299446 +0000 UTC" Nov 03 00:05:16.784361 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.784233 4304 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12827h23m10.923069582s" Nov 03 00:05:16.784648 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:16.784626 4304 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-172.us-west-1.compute.internal\" not found" Nov 03 00:05:16.785471 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785276 4304 factory.go:55] Registering systemd factory Nov 03 00:05:16.785471 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785313 4304 factory.go:223] Registration of the systemd container factory successfully Nov 03 00:05:16.785471 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785340 4304 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785594 4304 factory.go:153] Registering CRI-O factory Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785611 4304 factory.go:223] Registration of the crio container factory successfully Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785673 4304 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785599 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47d5d066-c799-4be0-9a2c-680351cb8095" volumeName="kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-binary-copy" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785691 4304 factory.go:103] Registering Raw factory Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785695 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47d5d066-c799-4be0-9a2c-680351cb8095" volumeName="kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785715 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0" volumeName="kubernetes.io/projected/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-kube-api-access-qq7js" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785727 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3fd5ced-e9c1-4b9a-9939-473318ebbd93" volumeName="kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-daemon-config" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785735 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3fd5ced-e9c1-4b9a-9939-473318ebbd93" volumeName="kubernetes.io/projected/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-kube-api-access-h4nfn" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785743 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed2418f7-c460-4057-bbc1-9d5f01278635" volumeName="kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-config" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785750 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed2418f7-c460-4057-bbc1-9d5f01278635" volumeName="kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-env-overrides" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785787 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa475895-c30a-4076-98c0-d941610c151f" volumeName="kubernetes.io/projected/fa475895-c30a-4076-98c0-d941610c151f-kube-api-access-glqv6" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785806 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47d5d066-c799-4be0-9a2c-680351cb8095" volumeName="kubernetes.io/projected/47d5d066-c799-4be0-9a2c-680351cb8095-kube-api-access-lk4dj" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785814 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0" volumeName="kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-tmp" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785821 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a058bf9-45a2-44fd-908f-0e8fc41c1e5b" volumeName="kubernetes.io/empty-dir/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-tmp-dir" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785829 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3fd5ced-e9c1-4b9a-9939-473318ebbd93" volumeName="kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cni-binary-copy" seLinuxMountContext="" Nov 03 00:05:16.785830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785837 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e744436f-8167-45f8-8e07-508d1cc24f27" volumeName="kubernetes.io/projected/e744436f-8167-45f8-8e07-508d1cc24f27-kube-api-access-fnq68" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785849 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="434003bf-315d-42b3-8ee4-9cdc53936d73" volumeName="kubernetes.io/configmap/434003bf-315d-42b3-8ee4-9cdc53936d73-konnectivity-ca" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785861 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0" volumeName="kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-tuned" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785869 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="825e190f-db64-4823-8b20-4b605ba2c336" volumeName="kubernetes.io/projected/825e190f-db64-4823-8b20-4b605ba2c336-kube-api-access-vhwlj" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785876 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e744436f-8167-45f8-8e07-508d1cc24f27" volumeName="kubernetes.io/configmap/e744436f-8167-45f8-8e07-508d1cc24f27-iptables-alerter-script" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785883 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad" volumeName="kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785890 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed2418f7-c460-4057-bbc1-9d5f01278635" volumeName="kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-script-lib" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785897 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed2418f7-c460-4057-bbc1-9d5f01278635" volumeName="kubernetes.io/projected/ed2418f7-c460-4057-bbc1-9d5f01278635-kube-api-access-m4ctz" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785904 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed2418f7-c460-4057-bbc1-9d5f01278635" volumeName="kubernetes.io/secret/ed2418f7-c460-4057-bbc1-9d5f01278635-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785720 4304 manager.go:1199] Started watching for new ooms in manager Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.785912 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="434003bf-315d-42b3-8ee4-9cdc53936d73" volumeName="kubernetes.io/secret/434003bf-315d-42b3-8ee4-9cdc53936d73-agent-certs" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.786010 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47d5d066-c799-4be0-9a2c-680351cb8095" volumeName="kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.786031 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a058bf9-45a2-44fd-908f-0e8fc41c1e5b" volumeName="kubernetes.io/projected/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-kube-api-access-4rwgs" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.786046 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc26ec7-d383-4580-a5c4-becaf9dc3278" volumeName="kubernetes.io/projected/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kube-api-access-7kjs8" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.786061 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9593d56b-b401-4c0b-80cc-21a9e034b75d" volumeName="kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.786072 4304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa475895-c30a-4076-98c0-d941610c151f" volumeName="kubernetes.io/configmap/fa475895-c30a-4076-98c0-d941610c151f-serviceca" seLinuxMountContext="" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.786083 4304 reconstruct.go:97] "Volume reconstruction finished" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.786089 4304 reconciler.go:29] "Reconciler: start to sync state" Nov 03 00:05:16.786594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.786496 4304 manager.go:319] Starting recovery of all containers Nov 03 00:05:16.812088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.812052 4304 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 00:05:16.821659 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.821630 4304 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 00:05:16.821659 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.821655 4304 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 00:05:16.821854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.821677 4304 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 00:05:16.821854 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:16.821718 4304 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 00:05:16.828456 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.828419 4304 generic.go:356] "Generic (PLEG): container finished" podID="d20bb628721d473930b8a876f613ab5c" containerID="ed299c83332a5167d8279eb390f4bfd629ab5979bf63f21c3153690d3485f7d4" exitCode=0 Nov 03 00:05:16.845211 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.845188 4304 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="d751717a3b457264054f636f89a6f9ffc9f1cbf321c175bf76423f007d44561a" exitCode=0 Nov 03 00:05:16.845211 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.845212 4304 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="1e47a9398ec658f9024c60bf40bba5435137f5319ea0ed96cc734de9a7ef1b53" exitCode=0 Nov 03 00:05:16.845370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.845221 4304 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="6dd432a8db1b1aca436d3e938be0d2075dd1c5485b02e3d47d7270287a3894a9" exitCode=0 Nov 03 00:05:16.845370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.845229 4304 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="decc529752103d346c0f22b3543778a8f272db748776c02db84bde47f7ca49cb" exitCode=0 Nov 03 00:05:16.845370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.845238 4304 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="3ee59459bf454a78999e57da19c0ef51523253f9506a8bfb62def0e521cb313c" exitCode=0 Nov 03 00:05:16.845370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.845245 4304 generic.go:356] "Generic (PLEG): container finished" podID="47d5d066-c799-4be0-9a2c-680351cb8095" containerID="500a1a8a36c8c959db627e2aa5e363272f1c1b9ac70bc971ed293776f40ad558" exitCode=0 Nov 03 00:05:16.845558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.845384 4304 manager.go:324] Recovery completed Nov 03 00:05:16.856926 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.856911 4304 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:16.857846 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.857831 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:16.857916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.857860 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:16.857916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.857872 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:16.859030 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859009 4304 cpu_manager.go:226] "Starting" policy="none" Nov 03 00:05:16.859030 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859022 4304 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 00:05:16.859030 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859025 4304 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:16.859166 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859037 4304 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 00:05:16.859166 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859143 4304 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 00:05:16.859223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859168 4304 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 00:05:16.859223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859183 4304 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 00:05:16.859223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859188 4304 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 00:05:16.859223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859194 4304 policy_none.go:50] "Start" Nov 03 00:05:16.859223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859202 4304 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 00:05:16.859223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859210 4304 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 00:05:16.859375 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859328 4304 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 00:05:16.859375 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859334 4304 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 00:05:16.859375 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859339 4304 policy_none.go:44] "Start" Nov 03 00:05:16.859747 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859732 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:16.859795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859761 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:16.859795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.859789 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:16.862962 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.862939 4304 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 00:05:16.865492 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.865482 4304 manager.go:341] "Starting Device Plugin manager" Nov 03 00:05:16.865547 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:16.865503 4304 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 00:05:16.865547 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.865511 4304 server.go:87] "Starting device plugin registration server" Nov 03 00:05:16.865739 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.865729 4304 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 00:05:16.865788 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.865740 4304 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 00:05:16.865840 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.865827 4304 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 00:05:16.865948 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.865920 4304 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 00:05:16.866032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.865958 4304 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 00:05:16.866032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.865967 4304 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 00:05:16.866119 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.866019 4304 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:05:16.865947633Z","UUID":"1afdc263-a5b3-4360-8db5-1e994d528700","Handler":null,"Name":"","Endpoint":""} Nov 03 00:05:16.867145 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:16.867124 4304 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 00:05:16.867249 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:16.867167 4304 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-172.us-west-1.compute.internal\" not found" Nov 03 00:05:16.867249 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.867206 4304 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 00:05:16.867249 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.867223 4304 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 00:05:16.922853 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.922353 4304 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal"] Nov 03 00:05:16.922944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.922895 4304 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:16.923701 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.923678 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:16.923803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.923706 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:16.923803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.923716 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:16.923934 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.923924 4304 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:16.924097 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924082 4304 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:16.924614 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924598 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:16.924614 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924604 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:16.924725 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924620 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:16.924725 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924635 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:16.924725 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924641 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:16.924725 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924655 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:16.924912 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:16.924753 4304 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-172.us-west-1.compute.internal\" not found" node="ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:16.924912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924863 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" event={"ID":"d20bb628721d473930b8a876f613ab5c","Type":"ContainerStarted","Data":"9b0974df247669d56cbee9c607370d2ba8ed1d6b5ce290d5c1d2fdf98b5ae376"} Nov 03 00:05:16.924912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924895 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" event={"ID":"d20bb628721d473930b8a876f613ab5c","Type":"ContainerDied","Data":"ed299c83332a5167d8279eb390f4bfd629ab5979bf63f21c3153690d3485f7d4"} Nov 03 00:05:16.924912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924908 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" event={"ID":"d20bb628721d473930b8a876f613ab5c","Type":"ContainerStarted","Data":"18f04304a2355eb8ec06ee6f09511152a167e79e98b977ac62db21d88e394809"} Nov 03 00:05:16.925052 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924960 4304 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:16.925052 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.924960 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" event={"ID":"3a634ffde00653c750ef568ec11d5617","Type":"ContainerStarted","Data":"22d0576dc690dd9691f0fa5758c339ea8d9e00e2816c698a8c13cbe313e364ab"} Nov 03 00:05:16.925128 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.925053 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" event={"ID":"3a634ffde00653c750ef568ec11d5617","Type":"ContainerStarted","Data":"d39c93f03caa50f35b40e92aedf5376c8cf5911d789f150e8cfaa2e77af45bd0"} Nov 03 00:05:16.925625 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.925610 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:16.925683 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.925635 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:16.925683 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.925644 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:16.925762 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:16.925751 4304 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-172.us-west-1.compute.internal\" not found" node="ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:16.966816 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.966796 4304 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:16.967466 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.967452 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:16.967520 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.967477 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:16.967520 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.967486 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:16.967520 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.967505 4304 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:16.980419 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.980397 4304 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:16.980419 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.980407 4304 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:16.980576 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:16.980560 4304 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 00:05:17.029242 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.029218 4304 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:17.078090 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.078059 4304 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:17.081584 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.081563 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.081652 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.081598 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.081652 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.081617 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a634ffde00653c750ef568ec11d5617-config\") pod \"kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"3a634ffde00653c750ef568ec11d5617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.086180 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.086163 4304 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.129463 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.129438 4304 nodeinfomanager.go:471] existing CSINode "ip-10-0-140-172.us-west-1.compute.internal" is owned by different node (oldNodeID="30608d69-5b7a-4981-8e3d-a610a65721b9", newNodeID=""), cleaning up... Nov 03 00:05:17.129516 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.129474 4304 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.129560 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.129525 4304 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.130100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.130086 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeNotReady" Nov 03 00:05:17.130161 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.130115 4304 setters.go:546] "Node became not ready" node="ip-10-0-140-172.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T00:05:17Z","lastTransitionTime":"2026-11-03T00:05:17Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 00:05:17.144340 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.144322 4304 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:17.144392 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:17.144356 4304 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.144392 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.144366 4304 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.182279 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.182229 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.182279 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.182254 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.182279 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.182272 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a634ffde00653c750ef568ec11d5617-config\") pod \"kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"3a634ffde00653c750ef568ec11d5617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.182415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.182314 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a634ffde00653c750ef568ec11d5617-config\") pod \"kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"3a634ffde00653c750ef568ec11d5617\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.182415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.182337 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.182415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.182340 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20bb628721d473930b8a876f613ab5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" (UID: \"d20bb628721d473930b8a876f613ab5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.199931 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.199910 4304 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:17.199998 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:17.199945 4304 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.774917 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.774885 4304 apiserver.go:52] "Watching apiserver" Nov 03 00:05:17.852417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.852387 4304 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.852566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.852550 4304 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.877568 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.877550 4304 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 00:05:17.877945 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.877927 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-bthfh","openshift-ovn-kubernetes/ovnkube-node-cwtgz","kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj","openshift-cluster-node-tuning-operator/tuned-d9dlt","openshift-image-registry/node-ca-nk4xm","openshift-multus/multus-additional-cni-plugins-kx7kh","openshift-network-operator/iptables-alerter-rc4t7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal","openshift-multus/multus-mmcgw","openshift-multus/network-metrics-daemon-zvfng","kube-system/global-pull-secret-syncer-dz9d5","kube-system/konnectivity-agent-rlgnn","openshift-dns/node-resolver-lk69m"] Nov 03 00:05:17.880437 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.880403 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:17.886625 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.886607 4304 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 00:05:17.886736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.886722 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-script-lib\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.886800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.886746 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.886839 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.886802 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-node-log\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.886839 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.886824 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-log-socket\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.886902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.886884 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.886961 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.886929 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/434003bf-315d-42b3-8ee4-9cdc53936d73-konnectivity-ca\") pod \"konnectivity-agent-rlgnn\" (UID: \"434003bf-315d-42b3-8ee4-9cdc53936d73\") " pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:05:17.886998 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.886985 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-socket-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.887034 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887006 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-registration-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.887034 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887022 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-slash\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.887102 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887036 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-system-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.887138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887098 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-multus\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.887138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887126 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.887223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887153 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-netd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.887273 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887248 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7js\" (UniqueName: \"kubernetes.io/projected/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-kube-api-access-qq7js\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.887324 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887281 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-system-cni-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.887324 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887311 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-var-lib-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.887397 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887335 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cnibin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.887397 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887361 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-kubelet-config\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:17.887469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887406 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-dbus\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:17.887498 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887471 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-env-overrides\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.887535 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887513 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-netns\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.887567 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887546 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-conf-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.887602 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887572 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwgs\" (UniqueName: \"kubernetes.io/projected/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-kube-api-access-4rwgs\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:05:17.887641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887600 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqv6\" (UniqueName: \"kubernetes.io/projected/fa475895-c30a-4076-98c0-d941610c151f-kube-api-access-glqv6\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:05:17.887641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887614 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e744436f-8167-45f8-8e07-508d1cc24f27-host-slash\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:05:17.887641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887628 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-kubelet\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.887793 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887673 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-tuned\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.887793 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887702 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa475895-c30a-4076-98c0-d941610c151f-serviceca\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:05:17.887793 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887758 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-tuned\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.887793 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887755 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/434003bf-315d-42b3-8ee4-9cdc53936d73-agent-certs\") pod \"konnectivity-agent-rlgnn\" (UID: \"434003bf-315d-42b3-8ee4-9cdc53936d73\") " pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:05:17.887957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887812 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-device-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.887957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887826 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-bin\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.887957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887849 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.887957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887874 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-etc-kubernetes\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.887957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887899 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-conf\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.887957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887914 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-systemd\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.887957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887931 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-tmp\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.887957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887946 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-systemd-units\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.888167 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887959 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-ovn\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.888167 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.887985 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nfn\" (UniqueName: \"kubernetes.io/projected/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-kube-api-access-h4nfn\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.888167 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888124 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:17.888167 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888143 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.888167 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888160 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa475895-c30a-4076-98c0-d941610c151f-host\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:05:17.888291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888175 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9kn\" (UniqueName: \"kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn\") pod \"network-check-target-bthfh\" (UID: \"9593d56b-b401-4c0b-80cc-21a9e034b75d\") " pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:17.888291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888203 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e744436f-8167-45f8-8e07-508d1cc24f27-iptables-alerter-script\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:05:17.888291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888217 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ctz\" (UniqueName: \"kubernetes.io/projected/ed2418f7-c460-4057-bbc1-9d5f01278635-kube-api-access-m4ctz\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.888291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888231 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cni-binary-copy\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.888291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888244 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-bin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.888291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888257 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.888291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888269 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-run\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.888291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888282 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-sys\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.888495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888328 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjs8\" (UniqueName: \"kubernetes.io/projected/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kube-api-access-7kjs8\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.888495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888344 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-daemon-config\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.888495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888359 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-kubernetes\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.888495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888373 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-lib-modules\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.888495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888387 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-host\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.888495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888400 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-cnibin\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.888495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888416 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4dj\" (UniqueName: \"kubernetes.io/projected/47d5d066-c799-4be0-9a2c-680351cb8095-kube-api-access-lk4dj\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.888495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888431 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwlj\" (UniqueName: \"kubernetes.io/projected/825e190f-db64-4823-8b20-4b605ba2c336-kube-api-access-vhwlj\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:17.888495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888452 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-sys-fs\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.888495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888470 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed2418f7-c460-4057-bbc1-9d5f01278635-ovn-node-metrics-cert\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.888495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888484 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-multus-certs\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888499 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-systemd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888562 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-socket-dir-parent\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888576 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-var-lib-kubelet\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888589 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-kubelet\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888604 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-hostroot\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888620 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-tmp-dir\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888640 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnq68\" (UniqueName: \"kubernetes.io/projected/e744436f-8167-45f8-8e07-508d1cc24f27-kube-api-access-fnq68\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888654 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-netns\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888672 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888687 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-k8s-cni-cncf-io\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888702 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysconfig\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888716 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888730 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888744 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-modprobe-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888758 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888790 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888819 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-etc-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.888827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888851 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.889299 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888867 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-config\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.889299 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.888881 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-hosts-file\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:05:17.889299 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.889053 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-tmp\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.889616 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.889594 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-tmp-dir\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:05:17.910434 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.910415 4304 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:17.910512 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:17.910455 4304 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.911268 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.911248 4304 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:17.911348 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:17.911298 4304 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-172.us-west-1.compute.internal" Nov 03 00:05:17.933185 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.933168 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 00:05:17.933356 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.933342 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 00:05:17.933504 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.933491 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 00:05:17.933594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.933578 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 00:05:17.933665 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.933618 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 00:05:17.933665 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.933654 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fmpf5\"" Nov 03 00:05:17.933757 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.933697 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:17.933852 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.933832 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7949\"" Nov 03 00:05:17.934046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.933935 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 00:05:17.934046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.933965 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 00:05:17.934046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.934008 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 00:05:17.934046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.934008 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 00:05:17.934046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.934029 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 00:05:17.934046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.934043 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z6zhf\"" Nov 03 00:05:17.934246 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.934073 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f67qw\"" Nov 03 00:05:17.934246 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.934121 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 00:05:17.934585 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.934572 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 00:05:17.936852 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.936809 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ptgp6\"" Nov 03 00:05:17.936932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.936875 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scmmg\"" Nov 03 00:05:17.936932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.936888 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8glgl\"" Nov 03 00:05:17.937065 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937048 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pkcxp\"" Nov 03 00:05:17.937122 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937113 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gd4nl\"" Nov 03 00:05:17.937172 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937137 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 00:05:17.937172 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937155 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 00:05:17.937172 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937162 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 00:05:17.937278 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937159 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 00:05:17.937278 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937190 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 00:05:17.937278 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937258 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nhzpn\"" Nov 03 00:05:17.937382 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937365 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.937427 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937382 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.937483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937430 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 00:05:17.937483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937428 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 00:05:17.937483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937444 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 00:05:17.937483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937450 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vqr8l\"" Nov 03 00:05:17.937483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937452 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 00:05:17.937483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937473 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 00:05:17.937803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937480 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 00:05:17.937803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937516 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 00:05:17.937803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937622 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 00:05:17.937803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937629 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 00:05:17.937803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937667 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-env-overrides\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.937803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937677 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 00:05:17.937803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.937731 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 00:05:17.938754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.938738 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/434003bf-315d-42b3-8ee4-9cdc53936d73-agent-certs\") pod \"konnectivity-agent-rlgnn\" (UID: \"434003bf-315d-42b3-8ee4-9cdc53936d73\") " pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:05:17.939139 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.939121 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa475895-c30a-4076-98c0-d941610c151f-serviceca\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:05:17.939342 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.939326 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-original-pull-secret\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:17.939410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.939345 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e744436f-8167-45f8-8e07-508d1cc24f27-iptables-alerter-script\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:05:17.939494 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.939482 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-daemon-config\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.939551 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.939485 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cni-binary-copy\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.939675 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.939660 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47d5d066-c799-4be0-9a2c-680351cb8095-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.939729 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.939660 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-config\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.939729 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.939711 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed2418f7-c460-4057-bbc1-9d5f01278635-ovn-node-metrics-cert\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.947923 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.947906 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/434003bf-315d-42b3-8ee4-9cdc53936d73-konnectivity-ca\") pod \"konnectivity-agent-rlgnn\" (UID: \"434003bf-315d-42b3-8ee4-9cdc53936d73\") " pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:05:17.947984 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.947951 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed2418f7-c460-4057-bbc1-9d5f01278635-ovnkube-script-lib\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.955448 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.955428 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:17.955542 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.955527 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 00:05:17.956463 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.956450 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k6628\"" Nov 03 00:05:17.960283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.960262 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nfn\" (UniqueName: \"kubernetes.io/projected/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-kube-api-access-h4nfn\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.960625 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.960611 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnq68\" (UniqueName: \"kubernetes.io/projected/e744436f-8167-45f8-8e07-508d1cc24f27-kube-api-access-fnq68\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:05:17.961460 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.961444 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rwgs\" (UniqueName: \"kubernetes.io/projected/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-kube-api-access-4rwgs\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:05:17.962167 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.962153 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7js\" (UniqueName: \"kubernetes.io/projected/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-kube-api-access-qq7js\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.962696 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.962678 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjs8\" (UniqueName: \"kubernetes.io/projected/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kube-api-access-7kjs8\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.962907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.962890 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqv6\" (UniqueName: \"kubernetes.io/projected/fa475895-c30a-4076-98c0-d941610c151f-kube-api-access-glqv6\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:05:17.963105 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.963091 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwlj\" (UniqueName: \"kubernetes.io/projected/825e190f-db64-4823-8b20-4b605ba2c336-kube-api-access-vhwlj\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:17.963860 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.963846 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ctz\" (UniqueName: \"kubernetes.io/projected/ed2418f7-c460-4057-bbc1-9d5f01278635-kube-api-access-m4ctz\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.964066 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.964046 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br9kn\" (UniqueName: \"kubernetes.io/projected/9593d56b-b401-4c0b-80cc-21a9e034b75d-kube-api-access-br9kn\") pod \"network-check-target-bthfh\" (UID: \"9593d56b-b401-4c0b-80cc-21a9e034b75d\") " pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:17.964478 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.964458 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4dj\" (UniqueName: \"kubernetes.io/projected/47d5d066-c799-4be0-9a2c-680351cb8095-kube-api-access-lk4dj\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.989233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989209 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-netns\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.989313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989235 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.989313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989252 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-k8s-cni-cncf-io\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.989313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989269 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysconfig\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.989313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989292 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989313 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-modprobe-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989323 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-netns\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989339 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989314 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989345 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-k8s-cni-cncf-io\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989372 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989396 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-etc-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989407 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989402 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989373 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysconfig\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989430 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-modprobe-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989422 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989461 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-etc-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.989487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989486 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-hosts-file\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:05:17.989919 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989518 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.989919 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989534 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-node-log\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.989919 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989552 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a058bf9-45a2-44fd-908f-0e8fc41c1e5b-hosts-file\") pod \"node-resolver-lk69m\" (UID: \"8a058bf9-45a2-44fd-908f-0e8fc41c1e5b\") " pod="openshift-dns/node-resolver-lk69m" Nov 03 00:05:17.989919 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989584 4304 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 00:05:17.989919 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.989628 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-node-log\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.990079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990027 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-log-socket\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.990079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990045 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.990079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990063 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-socket-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.990079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990076 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-registration-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990091 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-slash\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990104 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-system-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990117 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-multus\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990130 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-netd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990144 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-system-cni-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990159 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-var-lib-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990138 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990175 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cnibin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990209 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-log-socket\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990227 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-var-lib-openvswitch\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990238 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-socket-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990239 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-kubelet-config\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990269 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-slash\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990281 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-dbus\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990279 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-netd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990244 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-system-cni-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990313 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-netns\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990244 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-cnibin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990320 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-multus\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990283 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-registration-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990314 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-system-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990336 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-conf-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990359 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-netns\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.990353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990348 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-kubelet-config\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990378 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad-dbus\") pod \"global-pull-secret-syncer-dz9d5\" (UID: \"ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad\") " pod="kube-system/global-pull-secret-syncer-dz9d5" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990369 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-conf-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990371 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e744436f-8167-45f8-8e07-508d1cc24f27-host-slash\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990414 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e744436f-8167-45f8-8e07-508d1cc24f27-host-slash\") pod \"iptables-alerter-rc4t7\" (UID: \"e744436f-8167-45f8-8e07-508d1cc24f27\") " pod="openshift-network-operator/iptables-alerter-rc4t7" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990427 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-kubelet\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990453 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-device-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990469 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-bin\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990482 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990486 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-kubelet\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990496 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-etc-kubernetes\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990502 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-cni-bin\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990483 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-device-dir\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990525 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-conf\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990566 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-cni-dir\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990569 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-systemd\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990533 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-etc-kubernetes\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990597 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-systemd-units\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990609 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-systemd\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990628 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-ovn\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990637 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-systemd-units\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990650 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990664 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa475895-c30a-4076-98c0-d941610c151f-host\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990679 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-ovn\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990682 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-conf\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990680 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-bin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990708 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-var-lib-cni-bin\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990720 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa475895-c30a-4076-98c0-d941610c151f-host\") pod \"node-ca-nk4xm\" (UID: \"fa475895-c30a-4076-98c0-d941610c151f\") " pod="openshift-image-registry/node-ca-nk4xm" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990730 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990730 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990765 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-run\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990814 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-sys\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990820 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-run\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990832 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-kubernetes\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990848 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-lib-modules\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990852 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-sysctl-d\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990861 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-sys\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990861 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-host\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990898 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-cnibin\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990898 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-etc-kubernetes\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990912 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-sys-fs\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990927 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-multus-certs\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990934 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-lib-modules\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990940 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-host\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990948 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-systemd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990970 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47d5d066-c799-4be0-9a2c-680351cb8095-cnibin\") pod \"multus-additional-cni-plugins-kx7kh\" (UID: \"47d5d066-c799-4be0-9a2c-680351cb8095\") " pod="openshift-multus/multus-additional-cni-plugins-kx7kh" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990980 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8bc26ec7-d383-4580-a5c4-becaf9dc3278-sys-fs\") pod \"aws-ebs-csi-driver-node-rnccj\" (UID: \"8bc26ec7-d383-4580-a5c4-becaf9dc3278\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnccj" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.990995 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-host-run-multus-certs\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.991018 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-socket-dir-parent\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.991040 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-var-lib-kubelet\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.991044 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-run-systemd\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.991070 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0-var-lib-kubelet\") pod \"tuned-d9dlt\" (UID: \"81bb3e5f-3afc-4201-9ae7-3b3e8b0c27a0\") " pod="openshift-cluster-node-tuning-operator/tuned-d9dlt" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.991072 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-kubelet\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.991075 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-multus-socket-dir-parent\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.991098 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-hostroot\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.991105 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed2418f7-c460-4057-bbc1-9d5f01278635-host-kubelet\") pod \"ovnkube-node-cwtgz\" (UID: \"ed2418f7-c460-4057-bbc1-9d5f01278635\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:17.991187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.991140 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3fd5ced-e9c1-4b9a-9939-473318ebbd93-hostroot\") pod \"multus-mmcgw\" (UID: \"b3fd5ced-e9c1-4b9a-9939-473318ebbd93\") " pod="openshift-multus/multus-mmcgw" Nov 03 00:05:17.992789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:17.991823 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825e190f-db64-4823-8b20-4b605ba2c336-metrics-certs\") pod \"network-metrics-daemon-zvfng\" (UID: \"825e190f-db64-4823-8b20-4b605ba2c336\") " pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:18.181337 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:18.181305 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvfng" Nov 03 00:05:18.553074 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:18.552985 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zvfng"] Nov 03 00:05:18.556403 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:18.556378 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825e190f_db64_4823_8b20_4b605ba2c336.slice/crio-51665f5d8cac38769d81a67b112b189f775d551af4c96369c206a3081d18a75d WatchSource:0}: Error finding container 51665f5d8cac38769d81a67b112b189f775d551af4c96369c206a3081d18a75d: Status 404 returned error can't find the container with id 51665f5d8cac38769d81a67b112b189f775d551af4c96369c206a3081d18a75d Nov 03 00:05:18.558972 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:18.558957 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:05:18.651750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:18.651728 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:18.855310 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:18.855222 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zvfng" event={"ID":"825e190f-db64-4823-8b20-4b605ba2c336","Type":"ContainerStarted","Data":"51665f5d8cac38769d81a67b112b189f775d551af4c96369c206a3081d18a75d"} Nov 03 00:05:20.358207 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:20.358141 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:20.861272 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:20.861230 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zvfng" event={"ID":"825e190f-db64-4823-8b20-4b605ba2c336","Type":"ContainerStarted","Data":"00a06236c52aea95b0fd6171b2123cfc52461c2dc47f01c6acb4ba9a555c4c6b"} Nov 03 00:05:20.861420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:20.861292 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zvfng" event={"ID":"825e190f-db64-4823-8b20-4b605ba2c336","Type":"ContainerStarted","Data":"5485e460100c6fbb2c40b7615c310b857b56c32661310397aa8ab9eea771e39c"} Nov 03 00:05:21.202341 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:21.202316 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:21.202501 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:21.202489 4304 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:05:27.494028 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:27.494002 4304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-172.us-west-1.compute.internal" event="NodeReady" Nov 03 00:05:28.554242 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.554210 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22"] Nov 03 00:05:28.567045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.567022 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-hk9r4"] Nov 03 00:05:28.567156 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.567136 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22" Nov 03 00:05:28.583123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.583104 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c7fdc4d98-8292n"] Nov 03 00:05:28.583234 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.583218 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-hk9r4" Nov 03 00:05:28.597323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.597305 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.620222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.620202 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 00:05:28.620327 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.620250 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5w9jm\"" Nov 03 00:05:28.632430 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.632408 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xpdv8"] Nov 03 00:05:28.648095 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.648077 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.648095 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.648088 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xf5sc"] Nov 03 00:05:28.651499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.651472 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2c71cdcf-0223-41f4-8a03-a3776968ca02-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-w7r22\" (UID: \"2c71cdcf-0223-41f4-8a03-a3776968ca02\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22" Nov 03 00:05:28.651582 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.651513 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-service-ca\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.651582 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.651531 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkbnf\" (UniqueName: \"kubernetes.io/projected/ee8bffca-1562-4724-ae1f-15411eb71a4c-kube-api-access-dkbnf\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.651657 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.651619 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee8bffca-1562-4724-ae1f-15411eb71a4c-tmp\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.651657 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.651640 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-oauth-serving-cert\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.651739 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.651667 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-serving-cert\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.651739 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.651683 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e02ee3de-5821-4758-8833-583c1c216713-tmp\") pod \"downloads-6dd789c8b7-hk9r4\" (UID: \"e02ee3de-5821-4758-8833-583c1c216713\") " pod="openshift-console/downloads-6dd789c8b7-hk9r4" Nov 03 00:05:28.651739 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.651697 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26p6q\" (UniqueName: \"kubernetes.io/projected/e02ee3de-5821-4758-8833-583c1c216713-kube-api-access-26p6q\") pod \"downloads-6dd789c8b7-hk9r4\" (UID: \"e02ee3de-5821-4758-8833-583c1c216713\") " pod="openshift-console/downloads-6dd789c8b7-hk9r4" Nov 03 00:05:28.651877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.651820 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-oauth-config\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.651877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.651854 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-config\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.653301 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.653283 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 00:05:28.662169 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.662147 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c9c8679fc-5wmcs"] Nov 03 00:05:28.662285 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.662271 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 00:05:28.662330 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.662305 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xf5sc" Nov 03 00:05:28.668674 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.668649 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 00:05:28.668817 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.668702 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 00:05:28.668817 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.668707 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 00:05:28.674267 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.674250 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-898tb"] Nov 03 00:05:28.674392 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.674378 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.693922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.693896 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.699226 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.699192 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 00:05:28.717068 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.717047 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c7fdc4d98-8292n"] Nov 03 00:05:28.720117 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.720097 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-m4ktd\"" Nov 03 00:05:28.720259 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.720153 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 00:05:28.720259 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.720154 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bkhng\"" Nov 03 00:05:28.720259 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.720277 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p8dw8\"" Nov 03 00:05:28.720526 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.720309 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 00:05:28.720526 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.720510 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-hk9r4"] Nov 03 00:05:28.720616 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.720570 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jkmg8\"" Nov 03 00:05:28.720732 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.720718 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 00:05:28.721323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.721306 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 00:05:28.722562 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.722507 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 00:05:28.722658 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.722633 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 00:05:28.752220 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752194 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e02ee3de-5821-4758-8833-583c1c216713-tmp\") pod \"downloads-6dd789c8b7-hk9r4\" (UID: \"e02ee3de-5821-4758-8833-583c1c216713\") " pod="openshift-console/downloads-6dd789c8b7-hk9r4" Nov 03 00:05:28.752320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752222 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26p6q\" (UniqueName: \"kubernetes.io/projected/e02ee3de-5821-4758-8833-583c1c216713-kube-api-access-26p6q\") pod \"downloads-6dd789c8b7-hk9r4\" (UID: \"e02ee3de-5821-4758-8833-583c1c216713\") " pod="openshift-console/downloads-6dd789c8b7-hk9r4" Nov 03 00:05:28.752320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752243 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7858bff9-b4e6-4364-8324-f926cd83db4e-config-volume\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.752320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752260 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-oauth-config\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.752490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752339 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a95b05bd-1f6c-481c-945f-fa43ac14561d-trusted-ca\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.752490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752387 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkbnf\" (UniqueName: \"kubernetes.io/projected/ee8bffca-1562-4724-ae1f-15411eb71a4c-kube-api-access-dkbnf\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.752490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752414 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7858bff9-b4e6-4364-8324-f926cd83db4e-metrics-tls\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.752490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752443 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a95b05bd-1f6c-481c-945f-fa43ac14561d-image-registry-private-configuration\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.752490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752478 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njzkg\" (UniqueName: \"kubernetes.io/projected/7858bff9-b4e6-4364-8324-f926cd83db4e-kube-api-access-njzkg\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.752714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752507 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwns\" (UniqueName: \"kubernetes.io/projected/2c110df0-ca5e-435a-ad90-1c56a345a531-kube-api-access-wqwns\") pod \"ingress-canary-xf5sc\" (UID: \"2c110df0-ca5e-435a-ad90-1c56a345a531\") " pod="openshift-ingress-canary/ingress-canary-xf5sc" Nov 03 00:05:28.752714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752538 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-oauth-serving-cert\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.752714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752601 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddd27e23-9dc6-4a85-951c-031fd9d2e798-crio-socket\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.752714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752635 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e02ee3de-5821-4758-8833-583c1c216713-tmp\") pod \"downloads-6dd789c8b7-hk9r4\" (UID: \"e02ee3de-5821-4758-8833-583c1c216713\") " pod="openshift-console/downloads-6dd789c8b7-hk9r4" Nov 03 00:05:28.752714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752637 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7858bff9-b4e6-4364-8324-f926cd83db4e-tmp-dir\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.752714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752709 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-serving-cert\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.753123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752731 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a95b05bd-1f6c-481c-945f-fa43ac14561d-bound-sa-token\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.753123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752754 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-config\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.753123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752770 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd27e23-9dc6-4a85-951c-031fd9d2e798-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.753123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752916 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddd27e23-9dc6-4a85-951c-031fd9d2e798-data-volume\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.753123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.752989 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2c71cdcf-0223-41f4-8a03-a3776968ca02-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-w7r22\" (UID: \"2c71cdcf-0223-41f4-8a03-a3776968ca02\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22" Nov 03 00:05:28.753123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753020 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-service-ca\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.753123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753045 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c110df0-ca5e-435a-ad90-1c56a345a531-cert\") pod \"ingress-canary-xf5sc\" (UID: \"2c110df0-ca5e-435a-ad90-1c56a345a531\") " pod="openshift-ingress-canary/ingress-canary-xf5sc" Nov 03 00:05:28.753123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753074 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a95b05bd-1f6c-481c-945f-fa43ac14561d-ca-trust-extracted\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.753123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753097 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a95b05bd-1f6c-481c-945f-fa43ac14561d-registry-certificates\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.753123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753121 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v8zz\" (UniqueName: \"kubernetes.io/projected/a95b05bd-1f6c-481c-945f-fa43ac14561d-kube-api-access-4v8zz\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.753619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753150 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a95b05bd-1f6c-481c-945f-fa43ac14561d-installation-pull-secrets\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.753619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753174 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee8bffca-1562-4724-ae1f-15411eb71a4c-tmp\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.753619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753200 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ksb\" (UniqueName: \"kubernetes.io/projected/ddd27e23-9dc6-4a85-951c-031fd9d2e798-kube-api-access-99ksb\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.753619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753229 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd27e23-9dc6-4a85-951c-031fd9d2e798-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.753619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753255 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a95b05bd-1f6c-481c-945f-fa43ac14561d-registry-tls\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.753619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753317 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-oauth-serving-cert\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.753619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753443 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-config\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.753909 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753650 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee8bffca-1562-4724-ae1f-15411eb71a4c-tmp\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.753909 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.753872 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-service-ca\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.755108 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.755088 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-oauth-config\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.755185 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.755161 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-serving-cert\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.755498 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.755474 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2c71cdcf-0223-41f4-8a03-a3776968ca02-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-w7r22\" (UID: \"2c71cdcf-0223-41f4-8a03-a3776968ca02\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22" Nov 03 00:05:28.772966 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.770289 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 00:05:28.772966 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.770319 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 00:05:28.772966 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.770344 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 00:05:28.777003 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.773883 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 00:05:28.777003 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.775833 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-898tb"] Nov 03 00:05:28.777003 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.775943 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c9c8679fc-5wmcs"] Nov 03 00:05:28.780632 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.780610 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22"] Nov 03 00:05:28.783587 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.783567 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xpdv8"] Nov 03 00:05:28.811931 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.811869 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26p6q\" (UniqueName: \"kubernetes.io/projected/e02ee3de-5821-4758-8833-583c1c216713-kube-api-access-26p6q\") pod \"downloads-6dd789c8b7-hk9r4\" (UID: \"e02ee3de-5821-4758-8833-583c1c216713\") " pod="openshift-console/downloads-6dd789c8b7-hk9r4" Nov 03 00:05:28.812246 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.812229 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkbnf\" (UniqueName: \"kubernetes.io/projected/ee8bffca-1562-4724-ae1f-15411eb71a4c-kube-api-access-dkbnf\") pod \"console-7c7fdc4d98-8292n\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.820221 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.820198 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 00:05:28.820317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.820232 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 00:05:28.820317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.820239 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rdw2q\"" Nov 03 00:05:28.820317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.820230 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 00:05:28.820436 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.820342 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q5tjj\"" Nov 03 00:05:28.820436 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.820365 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 00:05:28.845837 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.845813 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xf5sc"] Nov 03 00:05:28.854443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854420 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c110df0-ca5e-435a-ad90-1c56a345a531-cert\") pod \"ingress-canary-xf5sc\" (UID: \"2c110df0-ca5e-435a-ad90-1c56a345a531\") " pod="openshift-ingress-canary/ingress-canary-xf5sc" Nov 03 00:05:28.854598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854450 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a95b05bd-1f6c-481c-945f-fa43ac14561d-ca-trust-extracted\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.854598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854473 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a95b05bd-1f6c-481c-945f-fa43ac14561d-registry-certificates\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.854598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854495 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v8zz\" (UniqueName: \"kubernetes.io/projected/a95b05bd-1f6c-481c-945f-fa43ac14561d-kube-api-access-4v8zz\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.854598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854520 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a95b05bd-1f6c-481c-945f-fa43ac14561d-installation-pull-secrets\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.854598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854543 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ksb\" (UniqueName: \"kubernetes.io/projected/ddd27e23-9dc6-4a85-951c-031fd9d2e798-kube-api-access-99ksb\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.854598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854567 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd27e23-9dc6-4a85-951c-031fd9d2e798-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.854916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854616 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a95b05bd-1f6c-481c-945f-fa43ac14561d-registry-tls\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.854916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854669 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7858bff9-b4e6-4364-8324-f926cd83db4e-config-volume\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.854916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854698 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a95b05bd-1f6c-481c-945f-fa43ac14561d-trusted-ca\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.854916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854728 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7858bff9-b4e6-4364-8324-f926cd83db4e-metrics-tls\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.854916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854757 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a95b05bd-1f6c-481c-945f-fa43ac14561d-image-registry-private-configuration\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.854916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854807 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njzkg\" (UniqueName: \"kubernetes.io/projected/7858bff9-b4e6-4364-8324-f926cd83db4e-kube-api-access-njzkg\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.854916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854830 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwns\" (UniqueName: \"kubernetes.io/projected/2c110df0-ca5e-435a-ad90-1c56a345a531-kube-api-access-wqwns\") pod \"ingress-canary-xf5sc\" (UID: \"2c110df0-ca5e-435a-ad90-1c56a345a531\") " pod="openshift-ingress-canary/ingress-canary-xf5sc" Nov 03 00:05:28.854916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854861 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddd27e23-9dc6-4a85-951c-031fd9d2e798-crio-socket\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.854916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854870 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a95b05bd-1f6c-481c-945f-fa43ac14561d-ca-trust-extracted\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.854916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854886 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7858bff9-b4e6-4364-8324-f926cd83db4e-tmp-dir\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.855359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854934 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a95b05bd-1f6c-481c-945f-fa43ac14561d-bound-sa-token\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.855359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854964 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd27e23-9dc6-4a85-951c-031fd9d2e798-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.855359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.854996 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddd27e23-9dc6-4a85-951c-031fd9d2e798-data-volume\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.855359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.855202 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7858bff9-b4e6-4364-8324-f926cd83db4e-tmp-dir\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.855359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.855260 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddd27e23-9dc6-4a85-951c-031fd9d2e798-data-volume\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.855359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.855275 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ddd27e23-9dc6-4a85-951c-031fd9d2e798-crio-socket\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.855639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.855375 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7858bff9-b4e6-4364-8324-f926cd83db4e-config-volume\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.855639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.855380 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ddd27e23-9dc6-4a85-951c-031fd9d2e798-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.856543 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.856238 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a95b05bd-1f6c-481c-945f-fa43ac14561d-registry-certificates\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.856543 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.856469 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a95b05bd-1f6c-481c-945f-fa43ac14561d-trusted-ca\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.857721 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.857694 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7858bff9-b4e6-4364-8324-f926cd83db4e-metrics-tls\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.857835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.857805 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c110df0-ca5e-435a-ad90-1c56a345a531-cert\") pod \"ingress-canary-xf5sc\" (UID: \"2c110df0-ca5e-435a-ad90-1c56a345a531\") " pod="openshift-ingress-canary/ingress-canary-xf5sc" Nov 03 00:05:28.857929 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.857906 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ddd27e23-9dc6-4a85-951c-031fd9d2e798-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.858048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.858025 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a95b05bd-1f6c-481c-945f-fa43ac14561d-installation-pull-secrets\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.858321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.858304 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a95b05bd-1f6c-481c-945f-fa43ac14561d-registry-tls\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.858373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.858328 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a95b05bd-1f6c-481c-945f-fa43ac14561d-image-registry-private-configuration\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.890209 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.890188 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22" Nov 03 00:05:28.905358 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.905338 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-hk9r4" Nov 03 00:05:28.911627 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.911605 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v8zz\" (UniqueName: \"kubernetes.io/projected/a95b05bd-1f6c-481c-945f-fa43ac14561d-kube-api-access-4v8zz\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.911755 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.911729 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a95b05bd-1f6c-481c-945f-fa43ac14561d-bound-sa-token\") pod \"image-registry-6c9c8679fc-5wmcs\" (UID: \"a95b05bd-1f6c-481c-945f-fa43ac14561d\") " pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:28.911939 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.911924 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njzkg\" (UniqueName: \"kubernetes.io/projected/7858bff9-b4e6-4364-8324-f926cd83db4e-kube-api-access-njzkg\") pod \"dns-default-xpdv8\" (UID: \"7858bff9-b4e6-4364-8324-f926cd83db4e\") " pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:28.912050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.912031 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwns\" (UniqueName: \"kubernetes.io/projected/2c110df0-ca5e-435a-ad90-1c56a345a531-kube-api-access-wqwns\") pod \"ingress-canary-xf5sc\" (UID: \"2c110df0-ca5e-435a-ad90-1c56a345a531\") " pod="openshift-ingress-canary/ingress-canary-xf5sc" Nov 03 00:05:28.912253 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.912235 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ksb\" (UniqueName: \"kubernetes.io/projected/ddd27e23-9dc6-4a85-951c-031fd9d2e798-kube-api-access-99ksb\") pod \"insights-runtime-extractor-898tb\" (UID: \"ddd27e23-9dc6-4a85-951c-031fd9d2e798\") " pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:28.920117 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.920105 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:28.987115 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:28.987087 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:29.007039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.006975 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:29.025358 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.025329 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xf5sc" Nov 03 00:05:29.087895 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.087864 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-898tb" Nov 03 00:05:29.293054 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.293021 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22"] Nov 03 00:05:29.296292 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:29.296268 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c71cdcf_0223_41f4_8a03_a3776968ca02.slice/crio-7de4b296e8474816c2103946bf8e18e1ade8ff17d5fc2668c9a4c20a25ac8d64 WatchSource:0}: Error finding container 7de4b296e8474816c2103946bf8e18e1ade8ff17d5fc2668c9a4c20a25ac8d64: Status 404 returned error can't find the container with id 7de4b296e8474816c2103946bf8e18e1ade8ff17d5fc2668c9a4c20a25ac8d64 Nov 03 00:05:29.343608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.343543 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-hk9r4"] Nov 03 00:05:29.346467 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:29.346446 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02ee3de_5821_4758_8833_583c1c216713.slice/crio-7c5e1e2a98654fa986c8855e28697cfc3a5d059699892d196ac30914b640f888 WatchSource:0}: Error finding container 7c5e1e2a98654fa986c8855e28697cfc3a5d059699892d196ac30914b640f888: Status 404 returned error can't find the container with id 7c5e1e2a98654fa986c8855e28697cfc3a5d059699892d196ac30914b640f888 Nov 03 00:05:29.430851 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.430818 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xpdv8"] Nov 03 00:05:29.431607 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.431556 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c7fdc4d98-8292n"] Nov 03 00:05:29.433271 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.433248 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c9c8679fc-5wmcs"] Nov 03 00:05:29.435365 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:29.435343 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8bffca_1562_4724_ae1f_15411eb71a4c.slice/crio-2710c22384f10edfe6ef46a30cee50eaabe3e318f8dc1bbc4b85960a5617e8e5 WatchSource:0}: Error finding container 2710c22384f10edfe6ef46a30cee50eaabe3e318f8dc1bbc4b85960a5617e8e5: Status 404 returned error can't find the container with id 2710c22384f10edfe6ef46a30cee50eaabe3e318f8dc1bbc4b85960a5617e8e5 Nov 03 00:05:29.436675 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:29.436652 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95b05bd_1f6c_481c_945f_fa43ac14561d.slice/crio-82a3c1dd77b7d305943d449c836c4f1c653676f03c5f7d09968fdcb1195b7bae WatchSource:0}: Error finding container 82a3c1dd77b7d305943d449c836c4f1c653676f03c5f7d09968fdcb1195b7bae: Status 404 returned error can't find the container with id 82a3c1dd77b7d305943d449c836c4f1c653676f03c5f7d09968fdcb1195b7bae Nov 03 00:05:29.444863 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.444823 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xf5sc"] Nov 03 00:05:29.447515 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:29.447488 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c110df0_ca5e_435a_ad90_1c56a345a531.slice/crio-e3bbcf23e92935d35f8b279452858866fdfb21a77adce07e2bc84504d0db6b72 WatchSource:0}: Error finding container e3bbcf23e92935d35f8b279452858866fdfb21a77adce07e2bc84504d0db6b72: Status 404 returned error can't find the container with id e3bbcf23e92935d35f8b279452858866fdfb21a77adce07e2bc84504d0db6b72 Nov 03 00:05:29.494090 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.494065 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-898tb"] Nov 03 00:05:29.496558 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:29.496535 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd27e23_9dc6_4a85_951c_031fd9d2e798.slice/crio-61a3c6af95992447f10800d843e8b184650cd0a71773d20bb437df0631a4d495 WatchSource:0}: Error finding container 61a3c6af95992447f10800d843e8b184650cd0a71773d20bb437df0631a4d495: Status 404 returned error can't find the container with id 61a3c6af95992447f10800d843e8b184650cd0a71773d20bb437df0631a4d495 Nov 03 00:05:29.883045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.882971 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-898tb" event={"ID":"ddd27e23-9dc6-4a85-951c-031fd9d2e798","Type":"ContainerStarted","Data":"66bc90d0161deeb532f4462c726f13c352a156d26af88c92046f927037ec1618"} Nov 03 00:05:29.883045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.883022 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-898tb" event={"ID":"ddd27e23-9dc6-4a85-951c-031fd9d2e798","Type":"ContainerStarted","Data":"61a3c6af95992447f10800d843e8b184650cd0a71773d20bb437df0631a4d495"} Nov 03 00:05:29.884347 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.884280 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xf5sc" event={"ID":"2c110df0-ca5e-435a-ad90-1c56a345a531","Type":"ContainerStarted","Data":"e3bbcf23e92935d35f8b279452858866fdfb21a77adce07e2bc84504d0db6b72"} Nov 03 00:05:29.889528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.889495 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7fdc4d98-8292n" event={"ID":"ee8bffca-1562-4724-ae1f-15411eb71a4c","Type":"ContainerStarted","Data":"2710c22384f10edfe6ef46a30cee50eaabe3e318f8dc1bbc4b85960a5617e8e5"} Nov 03 00:05:29.890828 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.890801 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22" event={"ID":"2c71cdcf-0223-41f4-8a03-a3776968ca02","Type":"ContainerStarted","Data":"7de4b296e8474816c2103946bf8e18e1ade8ff17d5fc2668c9a4c20a25ac8d64"} Nov 03 00:05:29.892323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.892297 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" event={"ID":"a95b05bd-1f6c-481c-945f-fa43ac14561d","Type":"ContainerStarted","Data":"0448d432ed08139c6c954065e280f5b3a69e2f66b329c180f45a52aad53fffd0"} Nov 03 00:05:29.892446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.892339 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" event={"ID":"a95b05bd-1f6c-481c-945f-fa43ac14561d","Type":"ContainerStarted","Data":"82a3c1dd77b7d305943d449c836c4f1c653676f03c5f7d09968fdcb1195b7bae"} Nov 03 00:05:29.892446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.892426 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:29.893533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.893498 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-hk9r4" event={"ID":"e02ee3de-5821-4758-8833-583c1c216713","Type":"ContainerStarted","Data":"7c5e1e2a98654fa986c8855e28697cfc3a5d059699892d196ac30914b640f888"} Nov 03 00:05:29.894614 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:29.894596 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xpdv8" event={"ID":"7858bff9-b4e6-4364-8324-f926cd83db4e","Type":"ContainerStarted","Data":"4c07f46a3c2b5612ef64bbd9b6c0da9c7c4efeab354573231fc592fe681f78ca"} Nov 03 00:05:30.009407 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:30.009350 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" podStartSLOduration=2.00933794 podStartE2EDuration="2.00933794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:30.00874548 +0000 UTC m=+13.292303165" watchObservedRunningTime="2026-11-03 00:05:30.00933794 +0000 UTC m=+13.292895608" Nov 03 00:05:31.905879 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:31.905840 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-898tb" event={"ID":"ddd27e23-9dc6-4a85-951c-031fd9d2e798","Type":"ContainerStarted","Data":"7a37f1e96c1529bd7ad5bc3364a02b6bbd32ca168efbe7230c6a49f188b82c50"} Nov 03 00:05:32.581358 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:32.581326 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lk69m_8a058bf9-45a2-44fd-908f-0e8fc41c1e5b/dns-node-resolver/0.log" Nov 03 00:05:32.929877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:32.929837 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22" event={"ID":"2c71cdcf-0223-41f4-8a03-a3776968ca02","Type":"ContainerStarted","Data":"0fe14d9d5d7b879e633196182d8cf199d81900c08006501e6256b52255886cad"} Nov 03 00:05:32.931350 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:32.931322 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22" Nov 03 00:05:32.937767 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:32.937740 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22" Nov 03 00:05:33.048315 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:33.047756 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-w7r22" podStartSLOduration=1.8963837319999999 podStartE2EDuration="5.047742303s" totalImagesPullingTime="3.151358571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:33.047241151 +0000 UTC m=+16.330798826" watchObservedRunningTime="2026-11-03 00:05:33.047742303 +0000 UTC m=+16.331299970" Nov 03 00:05:33.183458 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:33.183429 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c9c8679fc-5wmcs_a95b05bd-1f6c-481c-945f-fa43ac14561d/registry/0.log" Nov 03 00:05:33.937242 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:33.937199 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xpdv8" event={"ID":"7858bff9-b4e6-4364-8324-f926cd83db4e","Type":"ContainerStarted","Data":"d0925b05d9751515cdf553c6c45dee849a8609bceea4e4abbdf8e4e2321c01eb"} Nov 03 00:05:33.937702 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:33.937262 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xpdv8" event={"ID":"7858bff9-b4e6-4364-8324-f926cd83db4e","Type":"ContainerStarted","Data":"34546abfa7b1e47a0d3215a29e25b9fc35d20d5d67a45b22224f1fa7edee5056"} Nov 03 00:05:33.979983 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:33.979957 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nk4xm_fa475895-c30a-4076-98c0-d941610c151f/node-ca/0.log" Nov 03 00:05:34.050666 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:34.050608 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xpdv8" podStartSLOduration=2.588312445 podStartE2EDuration="6.050591687s" totalImagesPullingTime="3.462279242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:34.049636078 +0000 UTC m=+17.333193751" watchObservedRunningTime="2026-11-03 00:05:34.050591687 +0000 UTC m=+17.334149358" Nov 03 00:05:34.942977 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:34.942942 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-898tb" event={"ID":"ddd27e23-9dc6-4a85-951c-031fd9d2e798","Type":"ContainerStarted","Data":"85e7a9408f6d42b3f6cd003e7ed5cb97fd448cccf342ff8b0d73c5fe63ead8b3"} Nov 03 00:05:34.944572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:34.944539 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xf5sc" event={"ID":"2c110df0-ca5e-435a-ad90-1c56a345a531","Type":"ContainerStarted","Data":"b35f8cefbc5b608d97a8eb67f0407b854e6029e284f7bc81ae8a2530b49aad55"} Nov 03 00:05:34.944942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:34.944928 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:35.075673 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:35.075404 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-898tb" podStartSLOduration=2.032162661 podStartE2EDuration="7.075383303s" totalImagesPullingTime="5.043220642s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:05:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:35.074496678 +0000 UTC m=+18.358054369" watchObservedRunningTime="2026-11-03 00:05:35.075383303 +0000 UTC m=+18.358940977" Nov 03 00:05:35.186980 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:35.186937 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xf5sc" podStartSLOduration=1.8857063219999999 podStartE2EDuration="7.186922514s" totalImagesPullingTime="5.301216192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:35.186507119 +0000 UTC m=+18.470064792" watchObservedRunningTime="2026-11-03 00:05:35.186922514 +0000 UTC m=+18.470480187" Nov 03 00:05:35.954164 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:35.954121 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7fdc4d98-8292n" event={"ID":"ee8bffca-1562-4724-ae1f-15411eb71a4c","Type":"ContainerStarted","Data":"b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c"} Nov 03 00:05:36.067611 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:36.067560 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c7fdc4d98-8292n" podStartSLOduration=2.391825867 podStartE2EDuration="8.067542347s" totalImagesPullingTime="5.67571648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:36.066575459 +0000 UTC m=+19.350133133" watchObservedRunningTime="2026-11-03 00:05:36.067542347 +0000 UTC m=+19.351100016" Nov 03 00:05:38.920684 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:38.920645 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:38.921443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:38.920714 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:38.926410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:38.926387 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:38.968359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:38.968330 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:05:41.225024 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.224992 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w7s2c"] Nov 03 00:05:41.227659 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.227639 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.253634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.253608 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-wtmp\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.253769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.253643 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-tls\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.253769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.253664 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea3a3688-008d-4735-b23a-ef2bb50254b1-sys\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.253901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.253809 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-textfile\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.253901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.253849 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a3688-008d-4735-b23a-ef2bb50254b1-metrics-client-ca\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.253901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.253882 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjxl\" (UniqueName: \"kubernetes.io/projected/ea3a3688-008d-4735-b23a-ef2bb50254b1-kube-api-access-mqjxl\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.254047 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.253936 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea3a3688-008d-4735-b23a-ef2bb50254b1-root\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.254091 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.254049 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.254132 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.254089 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.279555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.279526 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 00:05:41.279707 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.279596 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 00:05:41.329064 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.329021 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 00:05:41.329343 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.329301 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 00:05:41.329519 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.329501 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 00:05:41.329598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.329538 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 00:05:41.329598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.329578 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d8jlp\"" Nov 03 00:05:41.355415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.354925 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a3688-008d-4735-b23a-ef2bb50254b1-metrics-client-ca\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.355415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.354989 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjxl\" (UniqueName: \"kubernetes.io/projected/ea3a3688-008d-4735-b23a-ef2bb50254b1-kube-api-access-mqjxl\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.355415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355015 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea3a3688-008d-4735-b23a-ef2bb50254b1-root\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.355415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355047 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.355415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355077 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.355415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355104 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea3a3688-008d-4735-b23a-ef2bb50254b1-root\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.355415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355111 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-wtmp\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.355415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355173 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-tls\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.355415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355200 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea3a3688-008d-4735-b23a-ef2bb50254b1-sys\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.355415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355227 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-wtmp\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.355415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355235 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-textfile\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.356025 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355570 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea3a3688-008d-4735-b23a-ef2bb50254b1-sys\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.356025 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355576 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea3a3688-008d-4735-b23a-ef2bb50254b1-metrics-client-ca\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.356025 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.355946 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-textfile\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.356406 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.356384 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.358200 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.358082 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-tls\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.358200 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.358109 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea3a3688-008d-4735-b23a-ef2bb50254b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.418033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.417998 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjxl\" (UniqueName: \"kubernetes.io/projected/ea3a3688-008d-4735-b23a-ef2bb50254b1-kube-api-access-mqjxl\") pod \"node-exporter-w7s2c\" (UID: \"ea3a3688-008d-4735-b23a-ef2bb50254b1\") " pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.540933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.540836 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w7s2c" Nov 03 00:05:41.551328 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:41.551298 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3a3688_008d_4735_b23a_ef2bb50254b1.slice/crio-5fe43151e23a9461abcc225791254c62ff737998f73764b779c3723466daf92c WatchSource:0}: Error finding container 5fe43151e23a9461abcc225791254c62ff737998f73764b779c3723466daf92c: Status 404 returned error can't find the container with id 5fe43151e23a9461abcc225791254c62ff737998f73764b779c3723466daf92c Nov 03 00:05:41.978738 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:41.978700 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7s2c" event={"ID":"ea3a3688-008d-4735-b23a-ef2bb50254b1","Type":"ContainerStarted","Data":"5fe43151e23a9461abcc225791254c62ff737998f73764b779c3723466daf92c"} Nov 03 00:05:43.551012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.550972 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 00:05:43.553491 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.553466 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.605714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.605678 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 00:05:43.605714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.605678 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 00:05:43.605979 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.605678 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 00:05:43.605979 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.605727 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k72nn\"" Nov 03 00:05:43.605979 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.605751 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 00:05:43.652709 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.652679 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rlgnn" Nov 03 00:05:43.655050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.655028 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 00:05:43.655185 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.655061 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 00:05:43.655185 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.655095 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 00:05:43.656366 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.656343 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 00:05:43.671969 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.671943 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672076 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.671984 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672076 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.672015 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.672077 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.672118 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-config-out\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.672142 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.672165 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672381 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.672206 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzz5q\" (UniqueName: \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-kube-api-access-mzz5q\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672381 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.672294 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-config-volume\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672381 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.672342 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-web-config\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672527 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.672403 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672527 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.672425 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.672527 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.672451 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.705647 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.705608 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 00:05:43.754711 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.754681 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773077 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-config-out\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773121 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773148 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773172 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzz5q\" (UniqueName: \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-kube-api-access-mzz5q\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773208 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-config-volume\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773234 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-web-config\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773261 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773284 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773322 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773352 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773385 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773425 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773452 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.773907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.773853 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.774839 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.774316 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.774839 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.774587 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.777611 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.777583 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.778190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.778148 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-config-out\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.778190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.778174 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.778357 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.778288 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.778407 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.778366 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.778407 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.778392 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-config-volume\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.779192 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.779152 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.779945 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.779896 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-web-config\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.786217 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.786197 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.830907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.830832 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzz5q\" (UniqueName: \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-kube-api-access-mzz5q\") pod \"alertmanager-main-0\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.869653 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.869612 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:05:43.988444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.988390 4304 generic.go:356] "Generic (PLEG): container finished" podID="ea3a3688-008d-4735-b23a-ef2bb50254b1" containerID="cc4763b921c3577158f468cec7ef4f9e51f2efea42035283b4aaa9b53dbbc482" exitCode=0 Nov 03 00:05:43.988597 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:43.988441 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7s2c" event={"ID":"ea3a3688-008d-4735-b23a-ef2bb50254b1","Type":"ContainerDied","Data":"cc4763b921c3577158f468cec7ef4f9e51f2efea42035283b4aaa9b53dbbc482"} Nov 03 00:05:44.303061 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:44.303025 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 00:05:44.306310 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:44.306283 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf941882_d56d_40a5_b640_4f10b532d6e3.slice/crio-0f3530fe7668966f9c996d196e9433e17291452857d131efbb5782019da18c64 WatchSource:0}: Error finding container 0f3530fe7668966f9c996d196e9433e17291452857d131efbb5782019da18c64: Status 404 returned error can't find the container with id 0f3530fe7668966f9c996d196e9433e17291452857d131efbb5782019da18c64 Nov 03 00:05:44.956525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:44.956492 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xpdv8" Nov 03 00:05:44.994562 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:44.994528 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7s2c" event={"ID":"ea3a3688-008d-4735-b23a-ef2bb50254b1","Type":"ContainerStarted","Data":"b69dbdbe87d6f7359ce9b5d9021e39a9f47f57c0f0d491057e7d36b27e27d521"} Nov 03 00:05:44.994800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:44.994592 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7s2c" event={"ID":"ea3a3688-008d-4735-b23a-ef2bb50254b1","Type":"ContainerStarted","Data":"8de4f3c7c59dc1812a6cc69cdaf365d3d7e0ec7a8100d0393e300cbe18631d65"} Nov 03 00:05:44.995886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:44.995865 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerStarted","Data":"0f3530fe7668966f9c996d196e9433e17291452857d131efbb5782019da18c64"} Nov 03 00:05:45.706455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:45.706416 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwtgz" Nov 03 00:05:45.876625 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:45.876564 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w7s2c" podStartSLOduration=3.375554435 podStartE2EDuration="4.876546835s" totalImagesPullingTime="1.417581475s" totalInitContainerRuntime="83.410925ms" isStatefulPod=true podCreationTimestamp="2026-11-03 00:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:45.185515908 +0000 UTC m=+28.469073583" watchObservedRunningTime="2026-11-03 00:05:45.876546835 +0000 UTC m=+29.160104508" Nov 03 00:05:47.006075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:47.006040 4304 generic.go:356] "Generic (PLEG): container finished" podID="df941882-d56d-40a5-b640-4f10b532d6e3" containerID="1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667" exitCode=0 Nov 03 00:05:47.006513 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:47.006083 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerDied","Data":"1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667"} Nov 03 00:05:47.235453 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:47.235419 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c7fdc4d98-8292n"] Nov 03 00:05:48.858697 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:48.858662 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bthfh" Nov 03 00:05:49.416231 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.416201 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-586d95488b-2w6wj"] Nov 03 00:05:49.418242 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.418218 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.519943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.519904 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-serving-cert\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.520320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.519960 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-console-config\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.520320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.520026 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-trusted-ca-bundle\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.520320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.520129 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-oauth-serving-cert\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.520320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.520180 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-service-ca\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.520320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.520278 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-oauth-config\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.520509 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.520357 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zspbt\" (UniqueName: \"kubernetes.io/projected/f16397a2-3a13-473c-80f5-7d9067f9835e-kube-api-access-zspbt\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.520509 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.520410 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16397a2-3a13-473c-80f5-7d9067f9835e-tmp\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.570207 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.570155 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 00:05:49.571219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.571194 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586d95488b-2w6wj"] Nov 03 00:05:49.621314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.621248 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-console-config\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.621314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.621283 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-trusted-ca-bundle\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.621496 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.621313 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-oauth-serving-cert\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.621496 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.621350 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-service-ca\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.621496 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.621380 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-oauth-config\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.621496 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.621430 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zspbt\" (UniqueName: \"kubernetes.io/projected/f16397a2-3a13-473c-80f5-7d9067f9835e-kube-api-access-zspbt\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.621496 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.621466 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16397a2-3a13-473c-80f5-7d9067f9835e-tmp\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.621708 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.621504 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-serving-cert\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.622135 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.622105 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-console-config\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.622293 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.622278 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-oauth-serving-cert\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.622339 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.622288 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-service-ca\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.622408 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.622384 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-trusted-ca-bundle\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.622471 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.622457 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16397a2-3a13-473c-80f5-7d9067f9835e-tmp\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.624419 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.624392 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-oauth-config\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.624739 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.624722 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-serving-cert\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.678449 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.678423 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zspbt\" (UniqueName: \"kubernetes.io/projected/f16397a2-3a13-473c-80f5-7d9067f9835e-kube-api-access-zspbt\") pod \"console-586d95488b-2w6wj\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:49.731020 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:49.730978 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:05:50.021411 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.021379 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-hk9r4" event={"ID":"e02ee3de-5821-4758-8833-583c1c216713","Type":"ContainerStarted","Data":"a7c74891acf5c37744f0f20c4ccafc3047f854b907542e273cfbd298c55e1ddc"} Nov 03 00:05:50.021837 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.021542 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-hk9r4" Nov 03 00:05:50.023837 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.023594 4304 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-hk9r4 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.7:8080/\": dial tcp 10.133.0.7:8080: connect: connection refused" start-of-body= Nov 03 00:05:50.023837 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.023641 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-hk9r4" podUID="e02ee3de-5821-4758-8833-583c1c216713" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.7:8080/\": dial tcp 10.133.0.7:8080: connect: connection refused" Nov 03 00:05:50.026370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.025743 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerStarted","Data":"7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8"} Nov 03 00:05:50.026370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.025851 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerStarted","Data":"0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3"} Nov 03 00:05:50.026370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.025899 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerStarted","Data":"6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55"} Nov 03 00:05:50.026370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.025942 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerStarted","Data":"30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91"} Nov 03 00:05:50.026370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.025983 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerStarted","Data":"dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39"} Nov 03 00:05:50.108949 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.108920 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586d95488b-2w6wj"] Nov 03 00:05:50.114504 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:50.114464 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16397a2_3a13_473c_80f5_7d9067f9835e.slice/crio-4e1b107e588f70505ebcf941dceabd46574ba920c8ef214f824008eb469c5f10 WatchSource:0}: Error finding container 4e1b107e588f70505ebcf941dceabd46574ba920c8ef214f824008eb469c5f10: Status 404 returned error can't find the container with id 4e1b107e588f70505ebcf941dceabd46574ba920c8ef214f824008eb469c5f10 Nov 03 00:05:50.136904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.136832 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-hk9r4" podStartSLOduration=1.625671437 podStartE2EDuration="22.136809218s" totalImagesPullingTime="20.511137781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:50.136191294 +0000 UTC m=+33.419748968" watchObservedRunningTime="2026-11-03 00:05:50.136809218 +0000 UTC m=+33.420366884" Nov 03 00:05:50.233811 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.233757 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:05:50.236895 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.236805 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.291086 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.291053 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 00:05:50.291308 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.291137 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9docuu3vh85sj\"" Nov 03 00:05:50.291308 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.291245 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 00:05:50.291496 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.291345 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 00:05:50.291496 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.291345 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 00:05:50.291746 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.291721 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hwj5b\"" Nov 03 00:05:50.292039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.291967 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 00:05:50.292039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.291999 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 00:05:50.292184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.292152 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 00:05:50.292242 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.292185 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 00:05:50.292293 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.292243 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 00:05:50.329379 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329347 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.329538 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329394 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.329538 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329415 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.329645 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329540 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.329645 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329567 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.329645 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329597 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.329883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329653 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6q6m\" (UniqueName: \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-kube-api-access-j6q6m\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.329883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329702 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.329883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329827 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.329883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329860 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-config-out\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.330072 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329895 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.330072 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.329955 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.330072 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.330064 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.330216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.330095 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.330216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.330119 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.330216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.330201 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-config\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.330346 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.330241 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.330346 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.330301 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.330419 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.330351 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.330419 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.330375 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-web-config\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.339666 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.339568 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 00:05:50.340985 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.340950 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 00:05:50.341149 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.341014 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 00:05:50.341149 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.341057 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 00:05:50.341366 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.341156 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 00:05:50.341366 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.341347 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 00:05:50.348154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.347959 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:05:50.431679 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.431638 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.431894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.431690 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.431894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.431744 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-config-out\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.431894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.431801 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.431894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.431830 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.431894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.431861 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.431894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.431888 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.431912 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.431950 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-config\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432085 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432124 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432160 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432186 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-web-config\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432212 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432254 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432279 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432308 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432332 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432370 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432400 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6q6m\" (UniqueName: \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-kube-api-access-j6q6m\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.432481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.432426 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.433170 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.433112 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.435556 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.434323 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.435556 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.434587 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.440520 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.440231 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.442263 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.441568 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.442263 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.442223 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.442408 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.442356 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.447605 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.447564 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-config\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.448125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.447958 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.449279 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.449239 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.462037 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.461986 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.462677 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.462654 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-web-config\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.463036 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.463010 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-config-out\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.463181 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.463162 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.463247 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.463158 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.463349 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.463300 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.463647 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.463627 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.465431 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.465378 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.489175 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.489147 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6q6m\" (UniqueName: \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-kube-api-access-j6q6m\") pod \"prometheus-k8s-0\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.566378 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.566278 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:05:50.903700 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:50.903667 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c9c8679fc-5wmcs" Nov 03 00:05:51.035343 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:51.035256 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerStarted","Data":"e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93"} Nov 03 00:05:51.037260 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:51.037231 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586d95488b-2w6wj" event={"ID":"f16397a2-3a13-473c-80f5-7d9067f9835e","Type":"ContainerStarted","Data":"92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98"} Nov 03 00:05:51.037404 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:51.037278 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586d95488b-2w6wj" event={"ID":"f16397a2-3a13-473c-80f5-7d9067f9835e","Type":"ContainerStarted","Data":"4e1b107e588f70505ebcf941dceabd46574ba920c8ef214f824008eb469c5f10"} Nov 03 00:05:51.047995 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:51.047970 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-hk9r4" Nov 03 00:05:51.061670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:51.061643 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:05:51.066675 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:05:51.066641 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf139a794_85c2_49fe_88e9_1297c1cd2586.slice/crio-5389b8debd2a80b60ad89c98d34b0a0bbd5b50ad72e7f2d0ecd34cce78058a3c WatchSource:0}: Error finding container 5389b8debd2a80b60ad89c98d34b0a0bbd5b50ad72e7f2d0ecd34cce78058a3c: Status 404 returned error can't find the container with id 5389b8debd2a80b60ad89c98d34b0a0bbd5b50ad72e7f2d0ecd34cce78058a3c Nov 03 00:05:51.159509 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:51.159300 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.4643701829999998 podStartE2EDuration="8.159281221s" totalImagesPullingTime="5.680317864s" totalInitContainerRuntime="14.593174ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:51.158152843 +0000 UTC m=+34.441710553" watchObservedRunningTime="2026-11-03 00:05:51.159281221 +0000 UTC m=+34.442838897" Nov 03 00:05:51.272812 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:51.272743 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-586d95488b-2w6wj" podStartSLOduration=2.272729581 podStartE2EDuration="2.272729581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:51.271709179 +0000 UTC m=+34.555266855" watchObservedRunningTime="2026-11-03 00:05:51.272729581 +0000 UTC m=+34.556287254" Nov 03 00:05:52.043495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:52.043455 4304 generic.go:356] "Generic (PLEG): container finished" podID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerID="9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61" exitCode=0 Nov 03 00:05:52.043946 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:52.043548 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerDied","Data":"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61"} Nov 03 00:05:52.043946 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:52.043621 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerStarted","Data":"5389b8debd2a80b60ad89c98d34b0a0bbd5b50ad72e7f2d0ecd34cce78058a3c"} Nov 03 00:05:57.069316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:57.069275 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerStarted","Data":"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03"} Nov 03 00:05:57.069806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:57.069358 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerStarted","Data":"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b"} Nov 03 00:05:57.782714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:57.782674 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586d95488b-2w6wj"] Nov 03 00:05:58.660120 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:05:58.660077 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:05:59.732019 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:05:59.731998 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:06:00.087686 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:00.087653 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerStarted","Data":"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa"} Nov 03 00:06:00.087878 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:00.087736 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerStarted","Data":"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f"} Nov 03 00:06:00.087878 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:00.087760 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerStarted","Data":"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743"} Nov 03 00:06:00.087878 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:00.087812 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerStarted","Data":"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34"} Nov 03 00:06:00.361304 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:00.361219 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.951665207 podStartE2EDuration="10.361201084s" totalImagesPullingTime="7.385751413s" totalInitContainerRuntime="23.784464ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:06:00.310892871 +0000 UTC m=+43.594450543" watchObservedRunningTime="2026-11-03 00:06:00.361201084 +0000 UTC m=+43.644758763" Nov 03 00:06:00.566405 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:00.566366 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:12.257030 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.256970 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c7fdc4d98-8292n" podUID="ee8bffca-1562-4724-ae1f-15411eb71a4c" containerName="console" containerID="cri-o://b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c" gracePeriod=15 Nov 03 00:06:12.525857 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.525792 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:06:12.624794 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.624747 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee8bffca-1562-4724-ae1f-15411eb71a4c-kube-api-access-dkbnf\" (UniqueName: \"kubernetes.io/projected/ee8bffca-1562-4724-ae1f-15411eb71a4c-kube-api-access-dkbnf\") pod \"ee8bffca-1562-4724-ae1f-15411eb71a4c\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " Nov 03 00:06:12.624972 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.624826 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-service-ca\") pod \"ee8bffca-1562-4724-ae1f-15411eb71a4c\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " Nov 03 00:06:12.624972 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.624862 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-config\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-config\") pod \"ee8bffca-1562-4724-ae1f-15411eb71a4c\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " Nov 03 00:06:12.624972 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.624918 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8bffca-1562-4724-ae1f-15411eb71a4c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee8bffca-1562-4724-ae1f-15411eb71a4c-tmp\") pod \"ee8bffca-1562-4724-ae1f-15411eb71a4c\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " Nov 03 00:06:12.624972 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.624964 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-oauth-serving-cert\") pod \"ee8bffca-1562-4724-ae1f-15411eb71a4c\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " Nov 03 00:06:12.625175 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.625006 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-serving-cert\") pod \"ee8bffca-1562-4724-ae1f-15411eb71a4c\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " Nov 03 00:06:12.625175 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.625041 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-oauth-config\") pod \"ee8bffca-1562-4724-ae1f-15411eb71a4c\" (UID: \"ee8bffca-1562-4724-ae1f-15411eb71a4c\") " Nov 03 00:06:12.625277 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.625245 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8bffca-1562-4724-ae1f-15411eb71a4c-tmp" pod "ee8bffca-1562-4724-ae1f-15411eb71a4c" (UID: "ee8bffca-1562-4724-ae1f-15411eb71a4c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:12.625389 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.625291 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-config" pod "ee8bffca-1562-4724-ae1f-15411eb71a4c" (UID: "ee8bffca-1562-4724-ae1f-15411eb71a4c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:12.625389 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.625295 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-service-ca" pod "ee8bffca-1562-4724-ae1f-15411eb71a4c" (UID: "ee8bffca-1562-4724-ae1f-15411eb71a4c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:12.625389 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.625335 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-oauth-serving-cert" pod "ee8bffca-1562-4724-ae1f-15411eb71a4c" (UID: "ee8bffca-1562-4724-ae1f-15411eb71a4c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:12.627258 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.627230 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8bffca-1562-4724-ae1f-15411eb71a4c-kube-api-access-dkbnf" pod "ee8bffca-1562-4724-ae1f-15411eb71a4c" (UID: "ee8bffca-1562-4724-ae1f-15411eb71a4c"). InnerVolumeSpecName "kube-api-access-dkbnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:12.627350 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.627276 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-serving-cert" pod "ee8bffca-1562-4724-ae1f-15411eb71a4c" (UID: "ee8bffca-1562-4724-ae1f-15411eb71a4c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:12.627350 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.627305 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-oauth-config" pod "ee8bffca-1562-4724-ae1f-15411eb71a4c" (UID: "ee8bffca-1562-4724-ae1f-15411eb71a4c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:12.726006 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.725980 4304 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-serving-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:12.726006 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.726004 4304 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-oauth-config\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:12.726188 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.726015 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkbnf\" (UniqueName: \"kubernetes.io/projected/ee8bffca-1562-4724-ae1f-15411eb71a4c-kube-api-access-dkbnf\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:12.726188 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.726028 4304 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:12.726188 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.726049 4304 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-console-config\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:12.726188 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.726057 4304 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee8bffca-1562-4724-ae1f-15411eb71a4c-tmp\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:12.726188 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:12.726067 4304 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee8bffca-1562-4724-ae1f-15411eb71a4c-oauth-serving-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:13.133077 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:13.133001 4304 generic.go:356] "Generic (PLEG): container finished" podID="ee8bffca-1562-4724-ae1f-15411eb71a4c" containerID="b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c" exitCode=0 Nov 03 00:06:13.133077 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:13.133054 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7fdc4d98-8292n" Nov 03 00:06:13.133307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:13.133093 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7fdc4d98-8292n" event={"ID":"ee8bffca-1562-4724-ae1f-15411eb71a4c","Type":"ContainerDied","Data":"b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c"} Nov 03 00:06:13.133307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:13.133153 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7fdc4d98-8292n" event={"ID":"ee8bffca-1562-4724-ae1f-15411eb71a4c","Type":"ContainerDied","Data":"2710c22384f10edfe6ef46a30cee50eaabe3e318f8dc1bbc4b85960a5617e8e5"} Nov 03 00:06:13.133307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:13.133179 4304 scope.go:122] "RemoveContainer" containerID="b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c" Nov 03 00:06:13.141078 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:13.141062 4304 scope.go:122] "RemoveContainer" containerID="b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c" Nov 03 00:06:13.141374 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:13.141358 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c\": container with ID starting with b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c not found: ID does not exist" containerID="b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c" Nov 03 00:06:13.141420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:13.141381 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c"} err="failed to get container status \"b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c\": rpc error: code = NotFound desc = could not find container \"b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c\": container with ID starting with b6bd68cd467b6d5c2a68f6fec317cd6375427202fe9de3a98af57e0ea020a48c not found: ID does not exist" Nov 03 00:06:13.294880 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:13.294848 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c7fdc4d98-8292n"] Nov 03 00:06:13.344024 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:13.343998 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c7fdc4d98-8292n"] Nov 03 00:06:14.826281 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:14.826248 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee8bffca-1562-4724-ae1f-15411eb71a4c" path="/var/lib/kubelet/pods/ee8bffca-1562-4724-ae1f-15411eb71a4c/volumes" Nov 03 00:06:22.808314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:22.808256 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-586d95488b-2w6wj" podUID="f16397a2-3a13-473c-80f5-7d9067f9835e" containerName="console" containerID="cri-o://92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98" gracePeriod=15 Nov 03 00:06:23.080557 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.080503 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:06:23.106715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.106687 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-trusted-ca-bundle\") pod \"f16397a2-3a13-473c-80f5-7d9067f9835e\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " Nov 03 00:06:23.106833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.106751 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16397a2-3a13-473c-80f5-7d9067f9835e-kube-api-access-zspbt\" (UniqueName: \"kubernetes.io/projected/f16397a2-3a13-473c-80f5-7d9067f9835e-kube-api-access-zspbt\") pod \"f16397a2-3a13-473c-80f5-7d9067f9835e\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " Nov 03 00:06:23.106833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.106802 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-oauth-serving-cert\") pod \"f16397a2-3a13-473c-80f5-7d9067f9835e\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " Nov 03 00:06:23.106940 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.106834 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-console-config\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-console-config\") pod \"f16397a2-3a13-473c-80f5-7d9067f9835e\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " Nov 03 00:06:23.106940 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.106853 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-serving-cert\") pod \"f16397a2-3a13-473c-80f5-7d9067f9835e\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " Nov 03 00:06:23.106940 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.106873 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16397a2-3a13-473c-80f5-7d9067f9835e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16397a2-3a13-473c-80f5-7d9067f9835e-tmp\") pod \"f16397a2-3a13-473c-80f5-7d9067f9835e\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " Nov 03 00:06:23.107079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.106971 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-oauth-config\") pod \"f16397a2-3a13-473c-80f5-7d9067f9835e\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " Nov 03 00:06:23.107079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.107016 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-trusted-ca-bundle" pod "f16397a2-3a13-473c-80f5-7d9067f9835e" (UID: "f16397a2-3a13-473c-80f5-7d9067f9835e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:23.107195 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.107108 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16397a2-3a13-473c-80f5-7d9067f9835e-tmp" pod "f16397a2-3a13-473c-80f5-7d9067f9835e" (UID: "f16397a2-3a13-473c-80f5-7d9067f9835e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:23.107247 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.107227 4304 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-trusted-ca-bundle\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:23.107295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.107249 4304 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16397a2-3a13-473c-80f5-7d9067f9835e-tmp\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:23.107295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.107250 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-oauth-serving-cert" pod "f16397a2-3a13-473c-80f5-7d9067f9835e" (UID: "f16397a2-3a13-473c-80f5-7d9067f9835e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:23.107295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.107276 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-console-config" pod "f16397a2-3a13-473c-80f5-7d9067f9835e" (UID: "f16397a2-3a13-473c-80f5-7d9067f9835e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:23.109058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.109037 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16397a2-3a13-473c-80f5-7d9067f9835e-kube-api-access-zspbt" pod "f16397a2-3a13-473c-80f5-7d9067f9835e" (UID: "f16397a2-3a13-473c-80f5-7d9067f9835e"). InnerVolumeSpecName "kube-api-access-zspbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:23.109146 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.109076 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-serving-cert" pod "f16397a2-3a13-473c-80f5-7d9067f9835e" (UID: "f16397a2-3a13-473c-80f5-7d9067f9835e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:23.109146 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.109094 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-oauth-config" pod "f16397a2-3a13-473c-80f5-7d9067f9835e" (UID: "f16397a2-3a13-473c-80f5-7d9067f9835e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:23.168176 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.168144 4304 generic.go:356] "Generic (PLEG): container finished" podID="f16397a2-3a13-473c-80f5-7d9067f9835e" containerID="92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98" exitCode=0 Nov 03 00:06:23.168274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.168190 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586d95488b-2w6wj" event={"ID":"f16397a2-3a13-473c-80f5-7d9067f9835e","Type":"ContainerDied","Data":"92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98"} Nov 03 00:06:23.168274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.168198 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586d95488b-2w6wj" Nov 03 00:06:23.168274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.168237 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586d95488b-2w6wj" event={"ID":"f16397a2-3a13-473c-80f5-7d9067f9835e","Type":"ContainerDied","Data":"4e1b107e588f70505ebcf941dceabd46574ba920c8ef214f824008eb469c5f10"} Nov 03 00:06:23.168274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.168264 4304 scope.go:122] "RemoveContainer" containerID="92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98" Nov 03 00:06:23.175862 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.175844 4304 scope.go:122] "RemoveContainer" containerID="92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98" Nov 03 00:06:23.176132 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:23.176114 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98\": container with ID starting with 92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98 not found: ID does not exist" containerID="92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98" Nov 03 00:06:23.176186 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.176146 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98"} err="failed to get container status \"92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98\": rpc error: code = NotFound desc = could not find container \"92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98\": container with ID starting with 92993b5289443f66e707d37f6e5898d833f913b1720a821057a2551301d44d98 not found: ID does not exist" Nov 03 00:06:23.207692 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.207649 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-service-ca\") pod \"f16397a2-3a13-473c-80f5-7d9067f9835e\" (UID: \"f16397a2-3a13-473c-80f5-7d9067f9835e\") " Nov 03 00:06:23.207833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.207821 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zspbt\" (UniqueName: \"kubernetes.io/projected/f16397a2-3a13-473c-80f5-7d9067f9835e-kube-api-access-zspbt\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:23.207894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.207834 4304 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-oauth-serving-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:23.207894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.207844 4304 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-console-config\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:23.207894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.207853 4304 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-serving-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:23.207894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.207868 4304 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f16397a2-3a13-473c-80f5-7d9067f9835e-console-oauth-config\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:23.208089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.208015 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-service-ca" pod "f16397a2-3a13-473c-80f5-7d9067f9835e" (UID: "f16397a2-3a13-473c-80f5-7d9067f9835e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:23.308323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.308295 4304 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f16397a2-3a13-473c-80f5-7d9067f9835e-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:23.585838 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.585805 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586d95488b-2w6wj"] Nov 03 00:06:23.635113 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:23.635083 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-586d95488b-2w6wj"] Nov 03 00:06:24.826902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:24.826872 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16397a2-3a13-473c-80f5-7d9067f9835e" path="/var/lib/kubelet/pods/f16397a2-3a13-473c-80f5-7d9067f9835e/volumes" Nov 03 00:06:28.042278 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:28.042238 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:28.085990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:28.085965 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:28.205263 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:28.205239 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:28.653952 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:28.653918 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:06:37.528982 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:37.528950 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 00:06:37.529398 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:37.529231 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="alertmanager" containerID="cri-o://dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39" gracePeriod=120 Nov 03 00:06:37.529398 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:37.529314 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="prom-label-proxy" containerID="cri-o://e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93" gracePeriod=120 Nov 03 00:06:37.529398 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:37.529302 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy-metric" containerID="cri-o://7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8" gracePeriod=120 Nov 03 00:06:37.529398 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:37.529352 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy-web" containerID="cri-o://6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55" gracePeriod=120 Nov 03 00:06:37.529580 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:37.529376 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="config-reloader" containerID="cri-o://30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91" gracePeriod=120 Nov 03 00:06:37.529580 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:37.529389 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy" containerID="cri-o://0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3" gracePeriod=120 Nov 03 00:06:38.228804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.228754 4304 generic.go:356] "Generic (PLEG): container finished" podID="df941882-d56d-40a5-b640-4f10b532d6e3" containerID="e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93" exitCode=0 Nov 03 00:06:38.228804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.228800 4304 generic.go:356] "Generic (PLEG): container finished" podID="df941882-d56d-40a5-b640-4f10b532d6e3" containerID="0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3" exitCode=0 Nov 03 00:06:38.228804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.228811 4304 generic.go:356] "Generic (PLEG): container finished" podID="df941882-d56d-40a5-b640-4f10b532d6e3" containerID="30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91" exitCode=0 Nov 03 00:06:38.229089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.228819 4304 generic.go:356] "Generic (PLEG): container finished" podID="df941882-d56d-40a5-b640-4f10b532d6e3" containerID="dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39" exitCode=0 Nov 03 00:06:38.229089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.228801 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerDied","Data":"e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93"} Nov 03 00:06:38.229089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.228929 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerDied","Data":"0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3"} Nov 03 00:06:38.229089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.228969 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerDied","Data":"30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91"} Nov 03 00:06:38.229089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.229004 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerDied","Data":"dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39"} Nov 03 00:06:38.808919 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.808878 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:38.916578 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916546 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-config-out\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.916578 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916583 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.916861 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916605 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-metrics-client-ca\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.916861 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916635 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-trusted-ca-bundle\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.916861 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916661 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-web\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.916861 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916690 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-cluster-tls-config\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.916861 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916723 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-main-db\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.916861 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916748 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-kube-api-access-mzz5q\" (UniqueName: \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-kube-api-access-mzz5q\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.916861 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916846 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-web-config\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-web-config\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.917180 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916884 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-config-volume\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-config-volume\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.917180 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916931 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-main-tls\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.917180 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.916996 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.917180 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.917050 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-tls-assets\" (UniqueName: \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-tls-assets\") pod \"df941882-d56d-40a5-b640-4f10b532d6e3\" (UID: \"df941882-d56d-40a5-b640-4f10b532d6e3\") " Nov 03 00:06:38.917180 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.917112 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-metrics-client-ca" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:38.917418 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.917251 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-trusted-ca-bundle" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:38.917418 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.917273 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-main-db" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:38.917418 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.917336 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-metrics-client-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:38.917418 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.917359 4304 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:38.919996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.919949 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-metric" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:38.920394 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.920370 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-config-out" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:38.920518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.920420 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-config-volume" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:38.920628 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.920604 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:38.920730 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.920700 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-kube-api-access-mzz5q" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "kube-api-access-mzz5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:38.920867 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.920792 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-tls-assets" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:38.921289 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.921266 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-web" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:38.921658 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.921628 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-main-tls" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:38.924352 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.924327 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-cluster-tls-config" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:38.931280 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:38.931258 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-web-config" pod "df941882-d56d-40a5-b640-4f10b532d6e3" (UID: "df941882-d56d-40a5-b640-4f10b532d6e3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:39.018367 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.018332 4304 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.018367 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.018365 4304 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-tls-assets\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.018367 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.018376 4304 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-config-out\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.018631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.018385 4304 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.018631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.018396 4304 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.018631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.018406 4304 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-cluster-tls-config\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.018631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.018415 4304 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df941882-d56d-40a5-b640-4f10b532d6e3-alertmanager-main-db\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.018631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.018423 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzz5q\" (UniqueName: \"kubernetes.io/projected/df941882-d56d-40a5-b640-4f10b532d6e3-kube-api-access-mzz5q\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.018631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.018433 4304 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-web-config\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.018631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.018440 4304 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.018631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.018449 4304 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df941882-d56d-40a5-b640-4f10b532d6e3-secret-alertmanager-main-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.235825 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.235730 4304 generic.go:356] "Generic (PLEG): container finished" podID="df941882-d56d-40a5-b640-4f10b532d6e3" containerID="7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8" exitCode=0 Nov 03 00:06:39.235825 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.235755 4304 generic.go:356] "Generic (PLEG): container finished" podID="df941882-d56d-40a5-b640-4f10b532d6e3" containerID="6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55" exitCode=0 Nov 03 00:06:39.235825 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.235798 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerDied","Data":"7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8"} Nov 03 00:06:39.236043 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.235841 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.236043 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.235859 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerDied","Data":"6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55"} Nov 03 00:06:39.236043 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.235878 4304 scope.go:122] "RemoveContainer" containerID="e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93" Nov 03 00:06:39.236043 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.235897 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df941882-d56d-40a5-b640-4f10b532d6e3","Type":"ContainerDied","Data":"0f3530fe7668966f9c996d196e9433e17291452857d131efbb5782019da18c64"} Nov 03 00:06:39.244204 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.244181 4304 scope.go:122] "RemoveContainer" containerID="7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8" Nov 03 00:06:39.251311 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.251287 4304 scope.go:122] "RemoveContainer" containerID="0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3" Nov 03 00:06:39.257750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.257733 4304 scope.go:122] "RemoveContainer" containerID="6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55" Nov 03 00:06:39.263990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.263972 4304 scope.go:122] "RemoveContainer" containerID="30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91" Nov 03 00:06:39.270661 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.270644 4304 scope.go:122] "RemoveContainer" containerID="dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39" Nov 03 00:06:39.277277 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.277242 4304 scope.go:122] "RemoveContainer" containerID="1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667" Nov 03 00:06:39.283641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.283625 4304 scope.go:122] "RemoveContainer" containerID="e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93" Nov 03 00:06:39.283912 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:39.283894 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93\": container with ID starting with e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93 not found: ID does not exist" containerID="e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93" Nov 03 00:06:39.283964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.283922 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93"} err="failed to get container status \"e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93\": rpc error: code = NotFound desc = could not find container \"e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93\": container with ID starting with e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93 not found: ID does not exist" Nov 03 00:06:39.283964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.283937 4304 scope.go:122] "RemoveContainer" containerID="7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8" Nov 03 00:06:39.284140 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:39.284126 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8\": container with ID starting with 7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8 not found: ID does not exist" containerID="7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8" Nov 03 00:06:39.284189 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.284142 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8"} err="failed to get container status \"7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8\": rpc error: code = NotFound desc = could not find container \"7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8\": container with ID starting with 7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8 not found: ID does not exist" Nov 03 00:06:39.284189 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.284153 4304 scope.go:122] "RemoveContainer" containerID="0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3" Nov 03 00:06:39.284379 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:39.284359 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3\": container with ID starting with 0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3 not found: ID does not exist" containerID="0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3" Nov 03 00:06:39.284429 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.284386 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3"} err="failed to get container status \"0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3\": rpc error: code = NotFound desc = could not find container \"0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3\": container with ID starting with 0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3 not found: ID does not exist" Nov 03 00:06:39.284429 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.284403 4304 scope.go:122] "RemoveContainer" containerID="6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55" Nov 03 00:06:39.284614 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:39.284600 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55\": container with ID starting with 6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55 not found: ID does not exist" containerID="6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55" Nov 03 00:06:39.284655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.284619 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55"} err="failed to get container status \"6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55\": rpc error: code = NotFound desc = could not find container \"6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55\": container with ID starting with 6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55 not found: ID does not exist" Nov 03 00:06:39.284655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.284631 4304 scope.go:122] "RemoveContainer" containerID="30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91" Nov 03 00:06:39.284909 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:39.284883 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91\": container with ID starting with 30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91 not found: ID does not exist" containerID="30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91" Nov 03 00:06:39.284987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.284920 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91"} err="failed to get container status \"30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91\": rpc error: code = NotFound desc = could not find container \"30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91\": container with ID starting with 30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91 not found: ID does not exist" Nov 03 00:06:39.284987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.284939 4304 scope.go:122] "RemoveContainer" containerID="dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39" Nov 03 00:06:39.285179 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:39.285164 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39\": container with ID starting with dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39 not found: ID does not exist" containerID="dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39" Nov 03 00:06:39.285232 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.285183 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39"} err="failed to get container status \"dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39\": rpc error: code = NotFound desc = could not find container \"dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39\": container with ID starting with dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39 not found: ID does not exist" Nov 03 00:06:39.285232 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.285195 4304 scope.go:122] "RemoveContainer" containerID="1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667" Nov 03 00:06:39.285387 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:39.285371 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667\": container with ID starting with 1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667 not found: ID does not exist" containerID="1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667" Nov 03 00:06:39.285450 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.285396 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667"} err="failed to get container status \"1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667\": rpc error: code = NotFound desc = could not find container \"1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667\": container with ID starting with 1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667 not found: ID does not exist" Nov 03 00:06:39.285450 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.285417 4304 scope.go:122] "RemoveContainer" containerID="e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93" Nov 03 00:06:39.285623 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.285605 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93"} err="failed to get container status \"e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93\": rpc error: code = NotFound desc = could not find container \"e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93\": container with ID starting with e68843544685eb5c4f5384fd0d954c891863186592152228e63462db763d2d93 not found: ID does not exist" Nov 03 00:06:39.285674 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.285624 4304 scope.go:122] "RemoveContainer" containerID="7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8" Nov 03 00:06:39.285827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.285809 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8"} err="failed to get container status \"7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8\": rpc error: code = NotFound desc = could not find container \"7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8\": container with ID starting with 7127140a6237d0661cc6cf61ac770670536b5f48ae7578de5b146bad13bbcef8 not found: ID does not exist" Nov 03 00:06:39.285881 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.285829 4304 scope.go:122] "RemoveContainer" containerID="0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3" Nov 03 00:06:39.286052 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.286036 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3"} err="failed to get container status \"0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3\": rpc error: code = NotFound desc = could not find container \"0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3\": container with ID starting with 0a1f8d9f034eb1a61cebe12c1c1dff202e791aa2039ae4e7b10c831a402fd4f3 not found: ID does not exist" Nov 03 00:06:39.286090 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.286054 4304 scope.go:122] "RemoveContainer" containerID="6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55" Nov 03 00:06:39.286262 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.286243 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55"} err="failed to get container status \"6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55\": rpc error: code = NotFound desc = could not find container \"6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55\": container with ID starting with 6451a823b531ebeba90d64688b0857b5cd0452f1e40d76a1c7396d0397abcd55 not found: ID does not exist" Nov 03 00:06:39.286305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.286262 4304 scope.go:122] "RemoveContainer" containerID="30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91" Nov 03 00:06:39.286441 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.286422 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91"} err="failed to get container status \"30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91\": rpc error: code = NotFound desc = could not find container \"30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91\": container with ID starting with 30f718b7068fb217e4a1a681393994709568c42a12b9f61d77eb5f701ecc1f91 not found: ID does not exist" Nov 03 00:06:39.286481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.286441 4304 scope.go:122] "RemoveContainer" containerID="dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39" Nov 03 00:06:39.286619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.286603 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39"} err="failed to get container status \"dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39\": rpc error: code = NotFound desc = could not find container \"dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39\": container with ID starting with dbf77a1224e0a25c14db02601c945f95c40415b0708a64dc9ab991183ef97f39 not found: ID does not exist" Nov 03 00:06:39.286655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.286620 4304 scope.go:122] "RemoveContainer" containerID="1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667" Nov 03 00:06:39.286841 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.286827 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667"} err="failed to get container status \"1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667\": rpc error: code = NotFound desc = could not find container \"1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667\": container with ID starting with 1891f902660a1daa6669f65ac621f5b7b374ac36b1588a51abd756ea98b2c667 not found: ID does not exist" Nov 03 00:06:39.400867 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.400829 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 00:06:39.404383 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:39.404362 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 00:06:40.827255 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:40.827220 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" path="/var/lib/kubelet/pods/df941882-d56d-40a5-b640-4f10b532d6e3/volumes" Nov 03 00:06:43.283815 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.283763 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:06:43.284672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.284159 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa" gracePeriod=600 Nov 03 00:06:43.284672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.284195 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="thanos-sidecar" containerID="cri-o://f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34" gracePeriod=600 Nov 03 00:06:43.284672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.284213 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy" containerID="cri-o://859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f" gracePeriod=600 Nov 03 00:06:43.284672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.284232 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="config-reloader" containerID="cri-o://97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03" gracePeriod=600 Nov 03 00:06:43.284672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.284155 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="prometheus" containerID="cri-o://3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b" gracePeriod=600 Nov 03 00:06:43.284672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.284245 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy-web" containerID="cri-o://41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743" gracePeriod=600 Nov 03 00:06:43.522076 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.522051 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:43.555262 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555189 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-config\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-config\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555262 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555235 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-tls-assets\" (UniqueName: \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-tls-assets\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555433 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555264 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-db\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555433 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555281 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-2\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555433 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555413 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-0\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555587 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555469 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-1\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555587 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555574 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-web-config\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-web-config\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555590 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-2" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:43.555690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555620 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-kube-rbac-proxy\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555657 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555683 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-grpc-tls\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555929 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555710 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-metrics-client-certs\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555929 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555762 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-tls\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555929 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555824 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555929 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555833 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-1" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:43.555929 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555856 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-kube-api-access-j6q6m\" (UniqueName: \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-kube-api-access-j6q6m\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555929 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555888 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-kubelet-serving-ca-bundle\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.555929 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555923 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-trusted-ca-bundle\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.556283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555953 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-metrics-client-ca\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.556283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.555983 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-serving-certs-ca-bundle\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.556283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.556023 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-thanos-prometheus-http-client-file\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.556283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.556054 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-config-out\") pod \"f139a794-85c2-49fe-88e9-1297c1cd2586\" (UID: \"f139a794-85c2-49fe-88e9-1297c1cd2586\") " Nov 03 00:06:43.556283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.556255 4304 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.556283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.556272 4304 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.556740 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.556290 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-db" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:43.556816 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.556746 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-kubelet-serving-ca-bundle" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:43.557239 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.557160 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-metrics-client-ca" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:43.557373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.557231 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-0" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:43.557438 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.557417 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-serving-certs-ca-bundle" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:43.557664 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.557638 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-trusted-ca-bundle" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:43.559742 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.559716 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-config-out" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:43.559742 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.559733 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-config" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:43.559959 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.559836 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-kube-rbac-proxy" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:43.560187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.560153 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-tls-assets" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:43.560326 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.560301 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-tls" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:43.560476 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.560447 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-metrics-client-certs" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:43.560709 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.560686 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-thanos-sidecar-tls" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:43.560810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.560712 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-kube-api-access-j6q6m" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "kube-api-access-j6q6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:43.560940 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.560925 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-grpc-tls" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:43.561378 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.561354 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-kube-rbac-proxy-web" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:43.561615 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.561597 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-thanos-prometheus-http-client-file" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:43.569184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.569159 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-web-config" pod "f139a794-85c2-49fe-88e9-1297c1cd2586" (UID: "f139a794-85c2-49fe-88e9-1297c1cd2586"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:43.657040 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657002 4304 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657040 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657039 4304 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-web-config\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657057 4304 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-kube-rbac-proxy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657072 4304 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657088 4304 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-grpc-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657101 4304 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-metrics-client-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657115 4304 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657130 4304 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657145 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6q6m\" (UniqueName: \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-kube-api-access-j6q6m\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657159 4304 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657173 4304 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657189 4304 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-metrics-client-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657203 4304 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f139a794-85c2-49fe-88e9-1297c1cd2586-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657216 4304 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657651 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657230 4304 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-config-out\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657651 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657243 4304 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f139a794-85c2-49fe-88e9-1297c1cd2586-config\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657651 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657257 4304 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f139a794-85c2-49fe-88e9-1297c1cd2586-tls-assets\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.657651 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:43.657270 4304 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f139a794-85c2-49fe-88e9-1297c1cd2586-prometheus-k8s-db\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:44.255206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255170 4304 generic.go:356] "Generic (PLEG): container finished" podID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerID="d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa" exitCode=0 Nov 03 00:06:44.255206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255200 4304 generic.go:356] "Generic (PLEG): container finished" podID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerID="859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f" exitCode=0 Nov 03 00:06:44.255206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255210 4304 generic.go:356] "Generic (PLEG): container finished" podID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerID="41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743" exitCode=0 Nov 03 00:06:44.255206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255217 4304 generic.go:356] "Generic (PLEG): container finished" podID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerID="f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34" exitCode=0 Nov 03 00:06:44.255206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255222 4304 generic.go:356] "Generic (PLEG): container finished" podID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerID="97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03" exitCode=0 Nov 03 00:06:44.255499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255229 4304 generic.go:356] "Generic (PLEG): container finished" podID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerID="3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b" exitCode=0 Nov 03 00:06:44.255499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255253 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerDied","Data":"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa"} Nov 03 00:06:44.255499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255320 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerDied","Data":"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f"} Nov 03 00:06:44.255499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255265 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.255499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255346 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerDied","Data":"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743"} Nov 03 00:06:44.255499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255370 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerDied","Data":"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34"} Nov 03 00:06:44.255499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255394 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerDied","Data":"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03"} Nov 03 00:06:44.255499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255414 4304 scope.go:122] "RemoveContainer" containerID="d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa" Nov 03 00:06:44.255499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255417 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerDied","Data":"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b"} Nov 03 00:06:44.255499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.255441 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f139a794-85c2-49fe-88e9-1297c1cd2586","Type":"ContainerDied","Data":"5389b8debd2a80b60ad89c98d34b0a0bbd5b50ad72e7f2d0ecd34cce78058a3c"} Nov 03 00:06:44.263084 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.263062 4304 scope.go:122] "RemoveContainer" containerID="859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f" Nov 03 00:06:44.270275 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.270255 4304 scope.go:122] "RemoveContainer" containerID="41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743" Nov 03 00:06:44.277126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.277106 4304 scope.go:122] "RemoveContainer" containerID="f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34" Nov 03 00:06:44.283799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.283759 4304 scope.go:122] "RemoveContainer" containerID="97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03" Nov 03 00:06:44.290688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.290526 4304 scope.go:122] "RemoveContainer" containerID="3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b" Nov 03 00:06:44.309508 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.309490 4304 scope.go:122] "RemoveContainer" containerID="9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61" Nov 03 00:06:44.316509 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.316489 4304 scope.go:122] "RemoveContainer" containerID="d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa" Nov 03 00:06:44.316768 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:44.316752 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": container with ID starting with d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa not found: ID does not exist" containerID="d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa" Nov 03 00:06:44.316885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.316864 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa"} err="failed to get container status \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": rpc error: code = NotFound desc = could not find container \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": container with ID starting with d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa not found: ID does not exist" Nov 03 00:06:44.316969 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.316887 4304 scope.go:122] "RemoveContainer" containerID="859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f" Nov 03 00:06:44.317160 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:44.317142 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": container with ID starting with 859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f not found: ID does not exist" containerID="859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f" Nov 03 00:06:44.317209 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.317167 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f"} err="failed to get container status \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": rpc error: code = NotFound desc = could not find container \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": container with ID starting with 859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f not found: ID does not exist" Nov 03 00:06:44.317209 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.317183 4304 scope.go:122] "RemoveContainer" containerID="41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743" Nov 03 00:06:44.317442 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:44.317424 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": container with ID starting with 41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743 not found: ID does not exist" containerID="41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743" Nov 03 00:06:44.317490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.317456 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743"} err="failed to get container status \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": rpc error: code = NotFound desc = could not find container \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": container with ID starting with 41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743 not found: ID does not exist" Nov 03 00:06:44.317490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.317470 4304 scope.go:122] "RemoveContainer" containerID="f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34" Nov 03 00:06:44.317720 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:44.317702 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": container with ID starting with f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34 not found: ID does not exist" containerID="f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34" Nov 03 00:06:44.317757 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.317725 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34"} err="failed to get container status \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": rpc error: code = NotFound desc = could not find container \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": container with ID starting with f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34 not found: ID does not exist" Nov 03 00:06:44.317757 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.317738 4304 scope.go:122] "RemoveContainer" containerID="97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03" Nov 03 00:06:44.317985 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:44.317970 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": container with ID starting with 97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03 not found: ID does not exist" containerID="97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03" Nov 03 00:06:44.318033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.317989 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03"} err="failed to get container status \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": rpc error: code = NotFound desc = could not find container \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": container with ID starting with 97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03 not found: ID does not exist" Nov 03 00:06:44.318033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.318002 4304 scope.go:122] "RemoveContainer" containerID="3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b" Nov 03 00:06:44.318230 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:44.318210 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": container with ID starting with 3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b not found: ID does not exist" containerID="3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b" Nov 03 00:06:44.318269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.318234 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b"} err="failed to get container status \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": rpc error: code = NotFound desc = could not find container \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": container with ID starting with 3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b not found: ID does not exist" Nov 03 00:06:44.318269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.318248 4304 scope.go:122] "RemoveContainer" containerID="9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61" Nov 03 00:06:44.318505 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:44.318491 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": container with ID starting with 9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61 not found: ID does not exist" containerID="9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61" Nov 03 00:06:44.318546 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.318512 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61"} err="failed to get container status \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": rpc error: code = NotFound desc = could not find container \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": container with ID starting with 9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61 not found: ID does not exist" Nov 03 00:06:44.318546 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.318530 4304 scope.go:122] "RemoveContainer" containerID="d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa" Nov 03 00:06:44.318743 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.318727 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa"} err="failed to get container status \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": rpc error: code = NotFound desc = could not find container \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": container with ID starting with d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa not found: ID does not exist" Nov 03 00:06:44.318743 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.318742 4304 scope.go:122] "RemoveContainer" containerID="859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f" Nov 03 00:06:44.319014 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.318998 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f"} err="failed to get container status \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": rpc error: code = NotFound desc = could not find container \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": container with ID starting with 859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f not found: ID does not exist" Nov 03 00:06:44.319068 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.319015 4304 scope.go:122] "RemoveContainer" containerID="41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743" Nov 03 00:06:44.319252 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.319233 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743"} err="failed to get container status \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": rpc error: code = NotFound desc = could not find container \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": container with ID starting with 41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743 not found: ID does not exist" Nov 03 00:06:44.319321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.319266 4304 scope.go:122] "RemoveContainer" containerID="f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34" Nov 03 00:06:44.319487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.319470 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34"} err="failed to get container status \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": rpc error: code = NotFound desc = could not find container \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": container with ID starting with f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34 not found: ID does not exist" Nov 03 00:06:44.319530 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.319487 4304 scope.go:122] "RemoveContainer" containerID="97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03" Nov 03 00:06:44.319737 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.319720 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03"} err="failed to get container status \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": rpc error: code = NotFound desc = could not find container \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": container with ID starting with 97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03 not found: ID does not exist" Nov 03 00:06:44.319824 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.319740 4304 scope.go:122] "RemoveContainer" containerID="3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b" Nov 03 00:06:44.320000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.319982 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b"} err="failed to get container status \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": rpc error: code = NotFound desc = could not find container \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": container with ID starting with 3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b not found: ID does not exist" Nov 03 00:06:44.320042 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.320000 4304 scope.go:122] "RemoveContainer" containerID="9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61" Nov 03 00:06:44.320208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.320189 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61"} err="failed to get container status \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": rpc error: code = NotFound desc = could not find container \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": container with ID starting with 9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61 not found: ID does not exist" Nov 03 00:06:44.320275 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.320209 4304 scope.go:122] "RemoveContainer" containerID="d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa" Nov 03 00:06:44.320431 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.320413 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa"} err="failed to get container status \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": rpc error: code = NotFound desc = could not find container \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": container with ID starting with d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa not found: ID does not exist" Nov 03 00:06:44.320497 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.320432 4304 scope.go:122] "RemoveContainer" containerID="859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f" Nov 03 00:06:44.320663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.320642 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f"} err="failed to get container status \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": rpc error: code = NotFound desc = could not find container \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": container with ID starting with 859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f not found: ID does not exist" Nov 03 00:06:44.320708 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.320664 4304 scope.go:122] "RemoveContainer" containerID="41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743" Nov 03 00:06:44.320914 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.320879 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743"} err="failed to get container status \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": rpc error: code = NotFound desc = could not find container \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": container with ID starting with 41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743 not found: ID does not exist" Nov 03 00:06:44.320964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.320915 4304 scope.go:122] "RemoveContainer" containerID="f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34" Nov 03 00:06:44.321138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.321122 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34"} err="failed to get container status \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": rpc error: code = NotFound desc = could not find container \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": container with ID starting with f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34 not found: ID does not exist" Nov 03 00:06:44.321138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.321137 4304 scope.go:122] "RemoveContainer" containerID="97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03" Nov 03 00:06:44.321368 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.321342 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03"} err="failed to get container status \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": rpc error: code = NotFound desc = could not find container \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": container with ID starting with 97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03 not found: ID does not exist" Nov 03 00:06:44.321415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.321368 4304 scope.go:122] "RemoveContainer" containerID="3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b" Nov 03 00:06:44.321606 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.321587 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b"} err="failed to get container status \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": rpc error: code = NotFound desc = could not find container \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": container with ID starting with 3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b not found: ID does not exist" Nov 03 00:06:44.321677 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.321617 4304 scope.go:122] "RemoveContainer" containerID="9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61" Nov 03 00:06:44.321851 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.321836 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61"} err="failed to get container status \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": rpc error: code = NotFound desc = could not find container \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": container with ID starting with 9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61 not found: ID does not exist" Nov 03 00:06:44.321901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.321851 4304 scope.go:122] "RemoveContainer" containerID="d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa" Nov 03 00:06:44.322068 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.322053 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa"} err="failed to get container status \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": rpc error: code = NotFound desc = could not find container \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": container with ID starting with d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa not found: ID does not exist" Nov 03 00:06:44.322109 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.322067 4304 scope.go:122] "RemoveContainer" containerID="859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f" Nov 03 00:06:44.322282 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.322265 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f"} err="failed to get container status \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": rpc error: code = NotFound desc = could not find container \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": container with ID starting with 859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f not found: ID does not exist" Nov 03 00:06:44.322319 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.322281 4304 scope.go:122] "RemoveContainer" containerID="41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743" Nov 03 00:06:44.322499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.322482 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743"} err="failed to get container status \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": rpc error: code = NotFound desc = could not find container \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": container with ID starting with 41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743 not found: ID does not exist" Nov 03 00:06:44.322541 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.322500 4304 scope.go:122] "RemoveContainer" containerID="f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34" Nov 03 00:06:44.322728 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.322711 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34"} err="failed to get container status \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": rpc error: code = NotFound desc = could not find container \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": container with ID starting with f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34 not found: ID does not exist" Nov 03 00:06:44.322768 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.322727 4304 scope.go:122] "RemoveContainer" containerID="97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03" Nov 03 00:06:44.322955 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.322936 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03"} err="failed to get container status \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": rpc error: code = NotFound desc = could not find container \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": container with ID starting with 97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03 not found: ID does not exist" Nov 03 00:06:44.322996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.322957 4304 scope.go:122] "RemoveContainer" containerID="3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b" Nov 03 00:06:44.323185 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.323170 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b"} err="failed to get container status \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": rpc error: code = NotFound desc = could not find container \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": container with ID starting with 3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b not found: ID does not exist" Nov 03 00:06:44.323231 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.323185 4304 scope.go:122] "RemoveContainer" containerID="9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61" Nov 03 00:06:44.323384 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.323367 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61"} err="failed to get container status \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": rpc error: code = NotFound desc = could not find container \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": container with ID starting with 9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61 not found: ID does not exist" Nov 03 00:06:44.323432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.323385 4304 scope.go:122] "RemoveContainer" containerID="d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa" Nov 03 00:06:44.323571 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.323552 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa"} err="failed to get container status \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": rpc error: code = NotFound desc = could not find container \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": container with ID starting with d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa not found: ID does not exist" Nov 03 00:06:44.323613 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.323571 4304 scope.go:122] "RemoveContainer" containerID="859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f" Nov 03 00:06:44.323749 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.323736 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f"} err="failed to get container status \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": rpc error: code = NotFound desc = could not find container \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": container with ID starting with 859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f not found: ID does not exist" Nov 03 00:06:44.323802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.323749 4304 scope.go:122] "RemoveContainer" containerID="41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743" Nov 03 00:06:44.324007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.323990 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743"} err="failed to get container status \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": rpc error: code = NotFound desc = could not find container \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": container with ID starting with 41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743 not found: ID does not exist" Nov 03 00:06:44.324059 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.324007 4304 scope.go:122] "RemoveContainer" containerID="f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34" Nov 03 00:06:44.324230 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.324213 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34"} err="failed to get container status \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": rpc error: code = NotFound desc = could not find container \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": container with ID starting with f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34 not found: ID does not exist" Nov 03 00:06:44.324230 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.324229 4304 scope.go:122] "RemoveContainer" containerID="97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03" Nov 03 00:06:44.324452 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.324438 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03"} err="failed to get container status \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": rpc error: code = NotFound desc = could not find container \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": container with ID starting with 97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03 not found: ID does not exist" Nov 03 00:06:44.324496 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.324453 4304 scope.go:122] "RemoveContainer" containerID="3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b" Nov 03 00:06:44.324672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.324650 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b"} err="failed to get container status \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": rpc error: code = NotFound desc = could not find container \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": container with ID starting with 3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b not found: ID does not exist" Nov 03 00:06:44.324721 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.324675 4304 scope.go:122] "RemoveContainer" containerID="9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61" Nov 03 00:06:44.324936 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.324916 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61"} err="failed to get container status \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": rpc error: code = NotFound desc = could not find container \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": container with ID starting with 9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61 not found: ID does not exist" Nov 03 00:06:44.324987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.324938 4304 scope.go:122] "RemoveContainer" containerID="d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa" Nov 03 00:06:44.325159 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.325141 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa"} err="failed to get container status \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": rpc error: code = NotFound desc = could not find container \"d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa\": container with ID starting with d8a30891a5c12a645271739b0fb9c0be9297fa97d49a65af3ce704fbb48e10aa not found: ID does not exist" Nov 03 00:06:44.325228 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.325161 4304 scope.go:122] "RemoveContainer" containerID="859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f" Nov 03 00:06:44.325407 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.325390 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f"} err="failed to get container status \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": rpc error: code = NotFound desc = could not find container \"859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f\": container with ID starting with 859630916d0c4483963f319f3c9e5ab26a0f2c43c3b18c8fa802bbec63f3995f not found: ID does not exist" Nov 03 00:06:44.325453 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.325408 4304 scope.go:122] "RemoveContainer" containerID="41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743" Nov 03 00:06:44.325654 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.325637 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743"} err="failed to get container status \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": rpc error: code = NotFound desc = could not find container \"41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743\": container with ID starting with 41cbb58f7583d994be544298dcc64650dee19835cda9e418c348a3a003e5c743 not found: ID does not exist" Nov 03 00:06:44.325697 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.325655 4304 scope.go:122] "RemoveContainer" containerID="f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34" Nov 03 00:06:44.325886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.325868 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34"} err="failed to get container status \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": rpc error: code = NotFound desc = could not find container \"f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34\": container with ID starting with f77242e35c9372b365a6486f139bce1b9217a02708f271a381a347fc652cee34 not found: ID does not exist" Nov 03 00:06:44.325935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.325886 4304 scope.go:122] "RemoveContainer" containerID="97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03" Nov 03 00:06:44.326088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.326070 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03"} err="failed to get container status \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": rpc error: code = NotFound desc = could not find container \"97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03\": container with ID starting with 97425f2df611a3e406507e8ea9d1d6352fc74ad023839275aff241ba7f932d03 not found: ID does not exist" Nov 03 00:06:44.326140 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.326088 4304 scope.go:122] "RemoveContainer" containerID="3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b" Nov 03 00:06:44.326321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.326304 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b"} err="failed to get container status \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": rpc error: code = NotFound desc = could not find container \"3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b\": container with ID starting with 3b2c7c7319f53a1995f1813db7e324a68274fe2a703a2e53e6cdf3729146f85b not found: ID does not exist" Nov 03 00:06:44.326366 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.326321 4304 scope.go:122] "RemoveContainer" containerID="9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61" Nov 03 00:06:44.326519 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.326500 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61"} err="failed to get container status \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": rpc error: code = NotFound desc = could not find container \"9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61\": container with ID starting with 9449ed12e381253b3f893578248bd1c733c6c08a64e8d78f8d9c01d130b6fc61 not found: ID does not exist" Nov 03 00:06:44.469751 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.469721 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:06:44.519591 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.519500 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:06:44.520138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520116 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:06:44.520373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520359 4304 cpu_manager.go:414] "containerMap: removing container" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy-metric" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520376 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy-metric" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520388 4304 cpu_manager.go:414] "containerMap: removing container" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="prom-label-proxy" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520396 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="prom-label-proxy" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520409 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="config-reloader" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520416 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="config-reloader" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520427 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="prometheus" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520434 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="prometheus" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520445 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="thanos-sidecar" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520451 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="thanos-sidecar" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520461 4304 cpu_manager.go:414] "containerMap: removing container" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy-web" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520469 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy-web" Nov 03 00:06:44.520534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520537 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520544 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520557 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy-web" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520563 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy-web" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520583 4304 cpu_manager.go:414] "containerMap: removing container" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="init-config-reloader" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520589 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="init-config-reloader" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520599 4304 cpu_manager.go:414] "containerMap: removing container" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="alertmanager" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520606 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="alertmanager" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520618 4304 cpu_manager.go:414] "containerMap: removing container" podUID="ee8bffca-1562-4724-ae1f-15411eb71a4c" containerName="console" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520625 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee8bffca-1562-4724-ae1f-15411eb71a4c" containerName="console" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520635 4304 cpu_manager.go:414] "containerMap: removing container" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="config-reloader" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520642 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="config-reloader" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520653 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy-thanos" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520660 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy-thanos" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520669 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f16397a2-3a13-473c-80f5-7d9067f9835e" containerName="console" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520677 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16397a2-3a13-473c-80f5-7d9067f9835e" containerName="console" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520690 4304 cpu_manager.go:414] "containerMap: removing container" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520697 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520706 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="init-config-reloader" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520713 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="init-config-reloader" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520796 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="config-reloader" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520809 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="prometheus" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520818 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="thanos-sidecar" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520826 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy-web" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520834 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520844 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy-web" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520853 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="alertmanager" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520862 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee8bffca-1562-4724-ae1f-15411eb71a4c" containerName="console" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520870 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="config-reloader" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520878 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" containerName="kube-rbac-proxy-thanos" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520886 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16397a2-3a13-473c-80f5-7d9067f9835e" containerName="console" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520902 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520911 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="kube-rbac-proxy-metric" Nov 03 00:06:44.521138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.520918 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="df941882-d56d-40a5-b640-4f10b532d6e3" containerName="prom-label-proxy" Nov 03 00:06:44.524633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.524613 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.563752 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.563714 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.563927 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.563757 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2d4dbde-f2df-4182-994e-aa841a180fa9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.563927 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.563805 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.563927 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.563894 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.563927 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.563923 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2d4dbde-f2df-4182-994e-aa841a180fa9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.563943 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.563969 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564021 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564067 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564120 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564143 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhskr\" (UniqueName: \"kubernetes.io/projected/f2d4dbde-f2df-4182-994e-aa841a180fa9-kube-api-access-qhskr\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564161 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564182 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564217 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564277 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564305 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564338 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564368 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564580 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564413 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-config\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.564580 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.564432 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.578050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578021 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 00:06:44.578050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578038 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 00:06:44.578269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578033 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 00:06:44.578269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578037 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 00:06:44.578269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578076 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 00:06:44.578269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578202 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 00:06:44.578269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578234 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 00:06:44.578269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578243 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 00:06:44.578269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578264 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 00:06:44.578269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578266 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 00:06:44.578639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578323 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 00:06:44.578639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578464 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hwj5b\"" Nov 03 00:06:44.578639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578468 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 00:06:44.578639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578484 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9docuu3vh85sj\"" Nov 03 00:06:44.578639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.578497 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 00:06:44.628910 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.628869 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 00:06:44.628910 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.628869 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 00:06:44.665305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665270 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665303 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665320 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665337 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665352 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665371 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-config\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665390 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665424 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665453 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2d4dbde-f2df-4182-994e-aa841a180fa9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665476 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665509 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665535 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2d4dbde-f2df-4182-994e-aa841a180fa9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665559 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665580 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665599 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665618 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665643 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665663 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhskr\" (UniqueName: \"kubernetes.io/projected/f2d4dbde-f2df-4182-994e-aa841a180fa9-kube-api-access-qhskr\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665689 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665723 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.665922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.665869 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.666439 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.666122 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.666439 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.666177 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.666552 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.666462 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.666552 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.666531 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.667161 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.667136 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.667612 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.667474 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.669313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.669103 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.669418 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.669390 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2d4dbde-f2df-4182-994e-aa841a180fa9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.669858 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.669806 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.669957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.669931 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.670658 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.670633 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.670883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.670862 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2d4dbde-f2df-4182-994e-aa841a180fa9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.670979 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.670924 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2d4dbde-f2df-4182-994e-aa841a180fa9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.671127 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.671086 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.671323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.671302 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.671443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.671426 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-config\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.671711 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.671694 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.672283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.672266 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2d4dbde-f2df-4182-994e-aa841a180fa9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.724323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.724295 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhskr\" (UniqueName: \"kubernetes.io/projected/f2d4dbde-f2df-4182-994e-aa841a180fa9-kube-api-access-qhskr\") pod \"prometheus-k8s-0\" (UID: \"f2d4dbde-f2df-4182-994e-aa841a180fa9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:44.757463 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.757435 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:06:44.826692 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.826623 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f139a794-85c2-49fe-88e9-1297c1cd2586" path="/var/lib/kubelet/pods/f139a794-85c2-49fe-88e9-1297c1cd2586/volumes" Nov 03 00:06:44.839072 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:44.839046 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:45.262601 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:45.262571 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 00:06:45.265836 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:06:45.265810 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d4dbde_f2df_4182_994e_aa841a180fa9.slice/crio-bd579985fcfef74deebcb9b3fafa8a481d245948d6449d130d6d48e5f4a1ff62 WatchSource:0}: Error finding container bd579985fcfef74deebcb9b3fafa8a481d245948d6449d130d6d48e5f4a1ff62: Status 404 returned error can't find the container with id bd579985fcfef74deebcb9b3fafa8a481d245948d6449d130d6d48e5f4a1ff62 Nov 03 00:06:46.263216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:46.263177 4304 generic.go:356] "Generic (PLEG): container finished" podID="f2d4dbde-f2df-4182-994e-aa841a180fa9" containerID="6f02be8eea899e0eecbe02af7294269794caa7f1ffc030f891420ac6502abd69" exitCode=0 Nov 03 00:06:46.263700 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:46.263239 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2d4dbde-f2df-4182-994e-aa841a180fa9","Type":"ContainerDied","Data":"6f02be8eea899e0eecbe02af7294269794caa7f1ffc030f891420ac6502abd69"} Nov 03 00:06:46.263700 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:46.263312 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2d4dbde-f2df-4182-994e-aa841a180fa9","Type":"ContainerStarted","Data":"bd579985fcfef74deebcb9b3fafa8a481d245948d6449d130d6d48e5f4a1ff62"} Nov 03 00:06:47.270905 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:47.270869 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2d4dbde-f2df-4182-994e-aa841a180fa9","Type":"ContainerStarted","Data":"8664530815c55c91aa4f44263f3dfc4c6969f1353a4d8d75d420bbe8af3b0200"} Nov 03 00:06:47.271322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:47.270960 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2d4dbde-f2df-4182-994e-aa841a180fa9","Type":"ContainerStarted","Data":"9005dc601e378330c43b8f521de79228f5f7572d864689a5496d76cad27fd162"} Nov 03 00:06:47.271322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:47.270986 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2d4dbde-f2df-4182-994e-aa841a180fa9","Type":"ContainerStarted","Data":"f36c1470b5803f8f5ac7c3467e59f81955531ce7e196d179f557956ac81d205f"} Nov 03 00:06:47.271322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:47.271010 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2d4dbde-f2df-4182-994e-aa841a180fa9","Type":"ContainerStarted","Data":"b5424734e4b0c5ea31c5818b53bb0e0e4ca53a41827ccaeb6d78c1a62e845894"} Nov 03 00:06:47.271322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:47.271033 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2d4dbde-f2df-4182-994e-aa841a180fa9","Type":"ContainerStarted","Data":"2992856d5305bd3b0e36037b31eccaff4cb0361f3c8c34b9f49d9bc641d745e0"} Nov 03 00:06:47.271322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:47.271056 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2d4dbde-f2df-4182-994e-aa841a180fa9","Type":"ContainerStarted","Data":"7d31d2713de96ae9b8a702311f17fed0ec7c6be37819cb53657c64823f4233b8"} Nov 03 00:06:47.394495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:47.394442 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.379250419 podStartE2EDuration="3.394428238s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.177819ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:06:47.393206315 +0000 UTC m=+90.676764012" watchObservedRunningTime="2026-11-03 00:06:47.394428238 +0000 UTC m=+90.677985950" Nov 03 00:06:49.839856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:06:49.839822 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:06:58.653387 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:06:58.653350 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:07:28.654305 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:07:28.654267 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:07:44.840048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:07:44.839957 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:07:44.855991 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:07:44.855960 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:07:45.480823 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:07:45.480796 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 00:07:58.653457 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:07:58.653421 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:08:07.849323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.849245 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m"] Nov 03 00:08:07.851252 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.851237 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:07.875217 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.875188 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ptt\" (UniqueName: \"kubernetes.io/projected/65087784-5de2-45e7-bade-f484f33a41ea-kube-api-access-l7ptt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:07.875328 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.875233 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:07.875367 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.875333 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:07.952088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.952061 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-2f9zg\"" Nov 03 00:08:07.952270 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.952094 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 00:08:07.952270 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.952149 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 00:08:07.953026 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.953007 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m"] Nov 03 00:08:07.975691 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.975665 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:07.975835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.975698 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:07.975835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.975754 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ptt\" (UniqueName: \"kubernetes.io/projected/65087784-5de2-45e7-bade-f484f33a41ea-kube-api-access-l7ptt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:07.976125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.976109 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:07.976164 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:07.976121 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:08.032458 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:08.032423 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ptt\" (UniqueName: \"kubernetes.io/projected/65087784-5de2-45e7-bade-f484f33a41ea-kube-api-access-l7ptt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:08.162532 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:08.162500 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:08.527553 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:08.527475 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m"] Nov 03 00:08:08.530476 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:08:08.530448 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65087784_5de2_45e7_bade_f484f33a41ea.slice/crio-14653d126dd7b9d498bee3e6beaa0344267ab69a0ae1e1dc14ea7c48fa5bc796 WatchSource:0}: Error finding container 14653d126dd7b9d498bee3e6beaa0344267ab69a0ae1e1dc14ea7c48fa5bc796: Status 404 returned error can't find the container with id 14653d126dd7b9d498bee3e6beaa0344267ab69a0ae1e1dc14ea7c48fa5bc796 Nov 03 00:08:08.541809 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:08.541754 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" event={"ID":"65087784-5de2-45e7-bade-f484f33a41ea","Type":"ContainerStarted","Data":"14653d126dd7b9d498bee3e6beaa0344267ab69a0ae1e1dc14ea7c48fa5bc796"} Nov 03 00:08:15.174810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.174606 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk"] Nov 03 00:08:15.178186 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.178162 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:15.207061 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.207038 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl"] Nov 03 00:08:15.210093 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.210078 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:15.227955 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.227929 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:15.228079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.227977 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:15.228124 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.228084 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzhf\" (UniqueName: \"kubernetes.io/projected/3a30d16c-4afd-4a8b-bba6-dc923bfea766-kube-api-access-9bzhf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:15.228166 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.228134 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:15.228210 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.228192 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:15.228250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.228227 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqg6\" (UniqueName: \"kubernetes.io/projected/8b6b8f9a-9476-43e6-b84a-62db03481a4f-kube-api-access-4bqg6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:15.255942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.255909 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-v7dc2\"" Nov 03 00:08:15.256687 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.256671 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr"] Nov 03 00:08:15.259749 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.259734 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk"] Nov 03 00:08:15.259847 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.259836 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:15.281753 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.281727 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k"] Nov 03 00:08:15.285481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.285456 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl"] Nov 03 00:08:15.285599 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.285564 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:15.293160 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.293139 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr"] Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330064 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lnbg\" (UniqueName: \"kubernetes.io/projected/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-kube-api-access-7lnbg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330147 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7977m\" (UniqueName: \"kubernetes.io/projected/84b805c7-75f8-4230-ae03-5ef7f752c2f2-kube-api-access-7977m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330198 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330245 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330279 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330308 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqg6\" (UniqueName: \"kubernetes.io/projected/8b6b8f9a-9476-43e6-b84a-62db03481a4f-kube-api-access-4bqg6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330343 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330395 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330444 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330485 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330514 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.330543 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzhf\" (UniqueName: \"kubernetes.io/projected/3a30d16c-4afd-4a8b-bba6-dc923bfea766-kube-api-access-9bzhf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.331445 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.331886 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.332427 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:15.334810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.332701 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:15.335648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.334862 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-9d2fz\"" Nov 03 00:08:15.337320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.337301 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-n4zlt\"" Nov 03 00:08:15.337875 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.337854 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k"] Nov 03 00:08:15.387183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.387154 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzhf\" (UniqueName: \"kubernetes.io/projected/3a30d16c-4afd-4a8b-bba6-dc923bfea766-kube-api-access-9bzhf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:15.387624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.387603 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqg6\" (UniqueName: \"kubernetes.io/projected/8b6b8f9a-9476-43e6-b84a-62db03481a4f-kube-api-access-4bqg6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:15.431495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.431416 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:15.431495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.431453 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:15.431495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.431470 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:15.431495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.431495 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lnbg\" (UniqueName: \"kubernetes.io/projected/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-kube-api-access-7lnbg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:15.431795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.431624 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7977m\" (UniqueName: \"kubernetes.io/projected/84b805c7-75f8-4230-ae03-5ef7f752c2f2-kube-api-access-7977m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:15.431795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.431677 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:15.431949 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.431923 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:15.432034 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.431927 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:15.432034 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.431929 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:15.432117 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.432038 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:15.487449 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.487419 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7977m\" (UniqueName: \"kubernetes.io/projected/84b805c7-75f8-4230-ae03-5ef7f752c2f2-kube-api-access-7977m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:15.487582 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.487554 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lnbg\" (UniqueName: \"kubernetes.io/projected/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-kube-api-access-7lnbg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:15.493364 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.493343 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:15.524519 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.524489 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:15.573519 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.570355 4304 generic.go:356] "Generic (PLEG): container finished" podID="65087784-5de2-45e7-bade-f484f33a41ea" containerID="58158de445173680f7dc1ed76a994451042c4c43654d5220395060ff6bef374c" exitCode=0 Nov 03 00:08:15.573519 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.570450 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" event={"ID":"65087784-5de2-45e7-bade-f484f33a41ea","Type":"ContainerDied","Data":"58158de445173680f7dc1ed76a994451042c4c43654d5220395060ff6bef374c"} Nov 03 00:08:15.575066 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.575048 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:15.601665 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.601638 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:15.903676 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.903653 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk"] Nov 03 00:08:15.905814 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:08:15.905770 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6b8f9a_9476_43e6_b84a_62db03481a4f.slice/crio-30d643681a8b91004e3aba6b9117264b8f15b0b5e02679369f5700e9dae2fa4b WatchSource:0}: Error finding container 30d643681a8b91004e3aba6b9117264b8f15b0b5e02679369f5700e9dae2fa4b: Status 404 returned error can't find the container with id 30d643681a8b91004e3aba6b9117264b8f15b0b5e02679369f5700e9dae2fa4b Nov 03 00:08:15.912415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.912389 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl"] Nov 03 00:08:15.915001 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:08:15.914978 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a30d16c_4afd_4a8b_bba6_dc923bfea766.slice/crio-78b76d1f634f99fb85ed28b7a8e46e1895f5d4dc32bd7768ce4ed4b3c7029cc8 WatchSource:0}: Error finding container 78b76d1f634f99fb85ed28b7a8e46e1895f5d4dc32bd7768ce4ed4b3c7029cc8: Status 404 returned error can't find the container with id 78b76d1f634f99fb85ed28b7a8e46e1895f5d4dc32bd7768ce4ed4b3c7029cc8 Nov 03 00:08:15.956448 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:15.956417 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr"] Nov 03 00:08:15.960761 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:08:15.960734 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b805c7_75f8_4230_ae03_5ef7f752c2f2.slice/crio-1f64a86c6779e749fa8f92fd5fedbd66595174036575febabb876bd670d86665 WatchSource:0}: Error finding container 1f64a86c6779e749fa8f92fd5fedbd66595174036575febabb876bd670d86665: Status 404 returned error can't find the container with id 1f64a86c6779e749fa8f92fd5fedbd66595174036575febabb876bd670d86665 Nov 03 00:08:16.004548 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.004521 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k"] Nov 03 00:08:16.011172 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:08:16.011130 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode504fb8b_9b06_4a3c_b557_d4eb6ecc8d7b.slice/crio-6d899d392c652c0cd64603f22e08510b7af10f6a8552b7e2db7d2285b74800d8 WatchSource:0}: Error finding container 6d899d392c652c0cd64603f22e08510b7af10f6a8552b7e2db7d2285b74800d8: Status 404 returned error can't find the container with id 6d899d392c652c0cd64603f22e08510b7af10f6a8552b7e2db7d2285b74800d8 Nov 03 00:08:16.575804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.575692 4304 generic.go:356] "Generic (PLEG): container finished" podID="84b805c7-75f8-4230-ae03-5ef7f752c2f2" containerID="7c546744d3bab559d849741ce9a87cb0316890ca20ea44e121f8d2b37c344bc1" exitCode=0 Nov 03 00:08:16.575804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.575790 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" event={"ID":"84b805c7-75f8-4230-ae03-5ef7f752c2f2","Type":"ContainerDied","Data":"7c546744d3bab559d849741ce9a87cb0316890ca20ea44e121f8d2b37c344bc1"} Nov 03 00:08:16.576320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.575844 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" event={"ID":"84b805c7-75f8-4230-ae03-5ef7f752c2f2","Type":"ContainerStarted","Data":"1f64a86c6779e749fa8f92fd5fedbd66595174036575febabb876bd670d86665"} Nov 03 00:08:16.577512 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.577484 4304 generic.go:356] "Generic (PLEG): container finished" podID="e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" containerID="7cd73c3da3db76e6220b39dd4872e4cd5e5d3391b7b3816fd0ec22310db5d1ca" exitCode=0 Nov 03 00:08:16.577647 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.577562 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" event={"ID":"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b","Type":"ContainerDied","Data":"7cd73c3da3db76e6220b39dd4872e4cd5e5d3391b7b3816fd0ec22310db5d1ca"} Nov 03 00:08:16.577647 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.577596 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" event={"ID":"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b","Type":"ContainerStarted","Data":"6d899d392c652c0cd64603f22e08510b7af10f6a8552b7e2db7d2285b74800d8"} Nov 03 00:08:16.579215 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.579192 4304 generic.go:356] "Generic (PLEG): container finished" podID="3a30d16c-4afd-4a8b-bba6-dc923bfea766" containerID="a2ca0178382e5d9c71c53675f23a9a99c23820da775b14b0c23fffbde685f3da" exitCode=0 Nov 03 00:08:16.579317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.579275 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" event={"ID":"3a30d16c-4afd-4a8b-bba6-dc923bfea766","Type":"ContainerDied","Data":"a2ca0178382e5d9c71c53675f23a9a99c23820da775b14b0c23fffbde685f3da"} Nov 03 00:08:16.579447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.579316 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" event={"ID":"3a30d16c-4afd-4a8b-bba6-dc923bfea766","Type":"ContainerStarted","Data":"78b76d1f634f99fb85ed28b7a8e46e1895f5d4dc32bd7768ce4ed4b3c7029cc8"} Nov 03 00:08:16.580822 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.580797 4304 generic.go:356] "Generic (PLEG): container finished" podID="8b6b8f9a-9476-43e6-b84a-62db03481a4f" containerID="0af7860e09734cf844467315b3edc15cd0cd1b42c9d46f7e19994285e646565b" exitCode=0 Nov 03 00:08:16.580945 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.580826 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" event={"ID":"8b6b8f9a-9476-43e6-b84a-62db03481a4f","Type":"ContainerDied","Data":"0af7860e09734cf844467315b3edc15cd0cd1b42c9d46f7e19994285e646565b"} Nov 03 00:08:16.580945 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:16.580868 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" event={"ID":"8b6b8f9a-9476-43e6-b84a-62db03481a4f","Type":"ContainerStarted","Data":"30d643681a8b91004e3aba6b9117264b8f15b0b5e02679369f5700e9dae2fa4b"} Nov 03 00:08:18.588681 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:18.588641 4304 generic.go:356] "Generic (PLEG): container finished" podID="3a30d16c-4afd-4a8b-bba6-dc923bfea766" containerID="318f544514a27f7544e5c01e14cd4f8d69f52f1e90bc559a2bc58600066667ca" exitCode=0 Nov 03 00:08:18.589118 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:18.588703 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" event={"ID":"3a30d16c-4afd-4a8b-bba6-dc923bfea766","Type":"ContainerDied","Data":"318f544514a27f7544e5c01e14cd4f8d69f52f1e90bc559a2bc58600066667ca"} Nov 03 00:08:18.590576 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:18.590551 4304 generic.go:356] "Generic (PLEG): container finished" podID="65087784-5de2-45e7-bade-f484f33a41ea" containerID="4d2d6cd0f0fe04b7f0d0cd9ac859396c07b585f7178d0aafc5f98aaf8e06babc" exitCode=0 Nov 03 00:08:18.590669 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:18.590593 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" event={"ID":"65087784-5de2-45e7-bade-f484f33a41ea","Type":"ContainerDied","Data":"4d2d6cd0f0fe04b7f0d0cd9ac859396c07b585f7178d0aafc5f98aaf8e06babc"} Nov 03 00:08:19.597450 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:19.597413 4304 generic.go:356] "Generic (PLEG): container finished" podID="8b6b8f9a-9476-43e6-b84a-62db03481a4f" containerID="c818744dcb1860f59e560c20195fc32178d5eb2b9fa7446bb1d25cffab9eda22" exitCode=0 Nov 03 00:08:19.597946 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:19.597522 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" event={"ID":"8b6b8f9a-9476-43e6-b84a-62db03481a4f","Type":"ContainerDied","Data":"c818744dcb1860f59e560c20195fc32178d5eb2b9fa7446bb1d25cffab9eda22"} Nov 03 00:08:19.600225 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:19.600024 4304 generic.go:356] "Generic (PLEG): container finished" podID="84b805c7-75f8-4230-ae03-5ef7f752c2f2" containerID="e39b3b03e7249137edcdc9678ef62f27fef513ca3a68448817b06465c83b4771" exitCode=0 Nov 03 00:08:19.600225 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:19.600158 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" event={"ID":"84b805c7-75f8-4230-ae03-5ef7f752c2f2","Type":"ContainerDied","Data":"e39b3b03e7249137edcdc9678ef62f27fef513ca3a68448817b06465c83b4771"} Nov 03 00:08:19.602567 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:19.602533 4304 generic.go:356] "Generic (PLEG): container finished" podID="e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" containerID="cd751c1a3266c50a57eb80fddc508c9869d24a82da0ca51f4c1f13df449db853" exitCode=0 Nov 03 00:08:19.602666 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:19.602580 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" event={"ID":"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b","Type":"ContainerDied","Data":"cd751c1a3266c50a57eb80fddc508c9869d24a82da0ca51f4c1f13df449db853"} Nov 03 00:08:28.671454 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:08:28.671405 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:08:30.670472 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:30.670435 4304 generic.go:356] "Generic (PLEG): container finished" podID="3a30d16c-4afd-4a8b-bba6-dc923bfea766" containerID="8a96c763cb5ab75b52473091c3282b9b04047f39211d1655f25bbdc6bb833e57" exitCode=0 Nov 03 00:08:30.670963 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:30.670476 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" event={"ID":"3a30d16c-4afd-4a8b-bba6-dc923bfea766","Type":"ContainerDied","Data":"8a96c763cb5ab75b52473091c3282b9b04047f39211d1655f25bbdc6bb833e57"} Nov 03 00:08:30.674236 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:30.674206 4304 generic.go:356] "Generic (PLEG): container finished" podID="65087784-5de2-45e7-bade-f484f33a41ea" containerID="af758c1e36f40c5c38226c79c5bb332d5cba7c9bb05282a406b107a0b64a3821" exitCode=0 Nov 03 00:08:30.674366 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:30.674266 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" event={"ID":"65087784-5de2-45e7-bade-f484f33a41ea","Type":"ContainerDied","Data":"af758c1e36f40c5c38226c79c5bb332d5cba7c9bb05282a406b107a0b64a3821"} Nov 03 00:08:31.680286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.680251 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" event={"ID":"8b6b8f9a-9476-43e6-b84a-62db03481a4f","Type":"ContainerStarted","Data":"06ad1663eab46aba917173b3c61385d44ed78602234e072ade5c7996a4a49fdc"} Nov 03 00:08:31.682391 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.682365 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" event={"ID":"84b805c7-75f8-4230-ae03-5ef7f752c2f2","Type":"ContainerStarted","Data":"46d42071fc4367be4e2d525cb30d62ebd7d79bd61a02b076c9aee6d5e5f43e86"} Nov 03 00:08:31.684417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.684394 4304 generic.go:356] "Generic (PLEG): container finished" podID="e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" containerID="82d5142be2312debd28445d03fb7e7d47656fcd1550c8ddcb57dbd5ae971cee7" exitCode=0 Nov 03 00:08:31.684510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.684474 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" event={"ID":"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b","Type":"ContainerDied","Data":"82d5142be2312debd28445d03fb7e7d47656fcd1550c8ddcb57dbd5ae971cee7"} Nov 03 00:08:31.794896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.794850 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" podStartSLOduration=2.612832714 podStartE2EDuration="16.794834293s" totalImagesPullingTime="14.182001579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:31.79399547 +0000 UTC m=+195.077553158" watchObservedRunningTime="2026-11-03 00:08:31.794834293 +0000 UTC m=+195.078391968" Nov 03 00:08:31.810858 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.810831 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:31.833057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.833038 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:31.872025 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.871973 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-util\" (UniqueName: \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-util\") pod \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " Nov 03 00:08:31.872025 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.872012 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-util\" (UniqueName: \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-util\") pod \"65087784-5de2-45e7-bade-f484f33a41ea\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " Nov 03 00:08:31.872198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.872039 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-bundle\") pod \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " Nov 03 00:08:31.872198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.872094 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-bundle\" (UniqueName: \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-bundle\") pod \"65087784-5de2-45e7-bade-f484f33a41ea\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " Nov 03 00:08:31.872198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.872137 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65087784-5de2-45e7-bade-f484f33a41ea-kube-api-access-l7ptt\" (UniqueName: \"kubernetes.io/projected/65087784-5de2-45e7-bade-f484f33a41ea-kube-api-access-l7ptt\") pod \"65087784-5de2-45e7-bade-f484f33a41ea\" (UID: \"65087784-5de2-45e7-bade-f484f33a41ea\") " Nov 03 00:08:31.872198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.872169 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a30d16c-4afd-4a8b-bba6-dc923bfea766-kube-api-access-9bzhf\" (UniqueName: \"kubernetes.io/projected/3a30d16c-4afd-4a8b-bba6-dc923bfea766-kube-api-access-9bzhf\") pod \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\" (UID: \"3a30d16c-4afd-4a8b-bba6-dc923bfea766\") " Nov 03 00:08:31.872545 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.872518 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-bundle" pod "65087784-5de2-45e7-bade-f484f33a41ea" (UID: "65087784-5de2-45e7-bade-f484f33a41ea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:31.872545 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.872535 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-bundle" pod "3a30d16c-4afd-4a8b-bba6-dc923bfea766" (UID: "3a30d16c-4afd-4a8b-bba6-dc923bfea766"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:31.874482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.874452 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a30d16c-4afd-4a8b-bba6-dc923bfea766-kube-api-access-9bzhf" pod "3a30d16c-4afd-4a8b-bba6-dc923bfea766" (UID: "3a30d16c-4afd-4a8b-bba6-dc923bfea766"). InnerVolumeSpecName "kube-api-access-9bzhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:31.874572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.874461 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65087784-5de2-45e7-bade-f484f33a41ea-kube-api-access-l7ptt" pod "65087784-5de2-45e7-bade-f484f33a41ea" (UID: "65087784-5de2-45e7-bade-f484f33a41ea"). InnerVolumeSpecName "kube-api-access-l7ptt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:31.876985 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.876966 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-util" pod "3a30d16c-4afd-4a8b-bba6-dc923bfea766" (UID: "3a30d16c-4afd-4a8b-bba6-dc923bfea766"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:31.877513 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.877493 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-util" pod "65087784-5de2-45e7-bade-f484f33a41ea" (UID: "65087784-5de2-45e7-bade-f484f33a41ea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:31.907141 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.907090 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" podStartSLOduration=2.583464004 podStartE2EDuration="16.907076526s" totalImagesPullingTime="14.323612522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:31.906441031 +0000 UTC m=+195.189998718" watchObservedRunningTime="2026-11-03 00:08:31.907076526 +0000 UTC m=+195.190634200" Nov 03 00:08:31.972626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.972595 4304 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-util\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:31.972626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.972619 4304 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-util\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:31.972626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.972627 4304 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a30d16c-4afd-4a8b-bba6-dc923bfea766-bundle\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:31.972902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.972635 4304 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65087784-5de2-45e7-bade-f484f33a41ea-bundle\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:31.972902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.972643 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7ptt\" (UniqueName: \"kubernetes.io/projected/65087784-5de2-45e7-bade-f484f33a41ea-kube-api-access-l7ptt\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:31.972902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:31.972652 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bzhf\" (UniqueName: \"kubernetes.io/projected/3a30d16c-4afd-4a8b-bba6-dc923bfea766-kube-api-access-9bzhf\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:32.690126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.690086 4304 generic.go:356] "Generic (PLEG): container finished" podID="8b6b8f9a-9476-43e6-b84a-62db03481a4f" containerID="06ad1663eab46aba917173b3c61385d44ed78602234e072ade5c7996a4a49fdc" exitCode=0 Nov 03 00:08:32.690554 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.690174 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" event={"ID":"8b6b8f9a-9476-43e6-b84a-62db03481a4f","Type":"ContainerDied","Data":"06ad1663eab46aba917173b3c61385d44ed78602234e072ade5c7996a4a49fdc"} Nov 03 00:08:32.691875 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.691851 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" event={"ID":"65087784-5de2-45e7-bade-f484f33a41ea","Type":"ContainerDied","Data":"14653d126dd7b9d498bee3e6beaa0344267ab69a0ae1e1dc14ea7c48fa5bc796"} Nov 03 00:08:32.692012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.691896 4304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14653d126dd7b9d498bee3e6beaa0344267ab69a0ae1e1dc14ea7c48fa5bc796" Nov 03 00:08:32.692012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.691861 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67vr2m" Nov 03 00:08:32.693885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.693862 4304 generic.go:356] "Generic (PLEG): container finished" podID="84b805c7-75f8-4230-ae03-5ef7f752c2f2" containerID="46d42071fc4367be4e2d525cb30d62ebd7d79bd61a02b076c9aee6d5e5f43e86" exitCode=0 Nov 03 00:08:32.694002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.693941 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" event={"ID":"84b805c7-75f8-4230-ae03-5ef7f752c2f2","Type":"ContainerDied","Data":"46d42071fc4367be4e2d525cb30d62ebd7d79bd61a02b076c9aee6d5e5f43e86"} Nov 03 00:08:32.695673 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.695649 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" event={"ID":"3a30d16c-4afd-4a8b-bba6-dc923bfea766","Type":"ContainerDied","Data":"78b76d1f634f99fb85ed28b7a8e46e1895f5d4dc32bd7768ce4ed4b3c7029cc8"} Nov 03 00:08:32.695673 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.695671 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6k78xl" Nov 03 00:08:32.695849 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.695690 4304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78b76d1f634f99fb85ed28b7a8e46e1895f5d4dc32bd7768ce4ed4b3c7029cc8" Nov 03 00:08:32.817393 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.817366 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:32.881198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.881166 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-kube-api-access-7lnbg\" (UniqueName: \"kubernetes.io/projected/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-kube-api-access-7lnbg\") pod \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " Nov 03 00:08:32.881401 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.881248 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-util\" (UniqueName: \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-util\") pod \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " Nov 03 00:08:32.881401 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.881294 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-bundle\") pod \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\" (UID: \"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b\") " Nov 03 00:08:32.881865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.881836 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-bundle" pod "e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" (UID: "e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:32.883421 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.883399 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-kube-api-access-7lnbg" pod "e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" (UID: "e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b"). InnerVolumeSpecName "kube-api-access-7lnbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:32.888116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.888089 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-util" pod "e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" (UID: "e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:32.982060 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.981975 4304 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-bundle\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:32.982060 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.982004 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lnbg\" (UniqueName: \"kubernetes.io/projected/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-kube-api-access-7lnbg\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:32.982060 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:32.982014 4304 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b-util\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:33.701462 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.701420 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" Nov 03 00:08:33.701462 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.701437 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l5t9k" event={"ID":"e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b","Type":"ContainerDied","Data":"6d899d392c652c0cd64603f22e08510b7af10f6a8552b7e2db7d2285b74800d8"} Nov 03 00:08:33.701994 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.701487 4304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d899d392c652c0cd64603f22e08510b7af10f6a8552b7e2db7d2285b74800d8" Nov 03 00:08:33.825074 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.825049 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:33.850729 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.850706 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:33.888608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.888579 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-util\" (UniqueName: \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-util\") pod \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " Nov 03 00:08:33.888744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.888626 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-bundle\") pod \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " Nov 03 00:08:33.888744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.888651 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-bundle\") pod \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " Nov 03 00:08:33.888744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.888679 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-util\" (UniqueName: \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-util\") pod \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " Nov 03 00:08:33.888744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.888733 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b805c7-75f8-4230-ae03-5ef7f752c2f2-kube-api-access-7977m\" (UniqueName: \"kubernetes.io/projected/84b805c7-75f8-4230-ae03-5ef7f752c2f2-kube-api-access-7977m\") pod \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\" (UID: \"84b805c7-75f8-4230-ae03-5ef7f752c2f2\") " Nov 03 00:08:33.888970 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.888818 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6b8f9a-9476-43e6-b84a-62db03481a4f-kube-api-access-4bqg6\" (UniqueName: \"kubernetes.io/projected/8b6b8f9a-9476-43e6-b84a-62db03481a4f-kube-api-access-4bqg6\") pod \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\" (UID: \"8b6b8f9a-9476-43e6-b84a-62db03481a4f\") " Nov 03 00:08:33.889329 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.889297 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-bundle" pod "8b6b8f9a-9476-43e6-b84a-62db03481a4f" (UID: "8b6b8f9a-9476-43e6-b84a-62db03481a4f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:33.889446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.889419 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-bundle" pod "84b805c7-75f8-4230-ae03-5ef7f752c2f2" (UID: "84b805c7-75f8-4230-ae03-5ef7f752c2f2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:33.891097 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.891066 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6b8f9a-9476-43e6-b84a-62db03481a4f-kube-api-access-4bqg6" pod "8b6b8f9a-9476-43e6-b84a-62db03481a4f" (UID: "8b6b8f9a-9476-43e6-b84a-62db03481a4f"). InnerVolumeSpecName "kube-api-access-4bqg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:33.891488 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.891468 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b805c7-75f8-4230-ae03-5ef7f752c2f2-kube-api-access-7977m" pod "84b805c7-75f8-4230-ae03-5ef7f752c2f2" (UID: "84b805c7-75f8-4230-ae03-5ef7f752c2f2"). InnerVolumeSpecName "kube-api-access-7977m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:33.894091 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.894070 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-util" pod "84b805c7-75f8-4230-ae03-5ef7f752c2f2" (UID: "84b805c7-75f8-4230-ae03-5ef7f752c2f2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:33.894226 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.894205 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-util" pod "8b6b8f9a-9476-43e6-b84a-62db03481a4f" (UID: "8b6b8f9a-9476-43e6-b84a-62db03481a4f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:33.989624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.989539 4304 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-bundle\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:33.989624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.989570 4304 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b6b8f9a-9476-43e6-b84a-62db03481a4f-util\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:33.989624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.989583 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7977m\" (UniqueName: \"kubernetes.io/projected/84b805c7-75f8-4230-ae03-5ef7f752c2f2-kube-api-access-7977m\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:33.989624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.989594 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bqg6\" (UniqueName: \"kubernetes.io/projected/8b6b8f9a-9476-43e6-b84a-62db03481a4f-kube-api-access-4bqg6\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:33.989624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.989602 4304 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-util\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:33.989624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:33.989612 4304 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84b805c7-75f8-4230-ae03-5ef7f752c2f2-bundle\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:34.706560 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:34.706522 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" event={"ID":"8b6b8f9a-9476-43e6-b84a-62db03481a4f","Type":"ContainerDied","Data":"30d643681a8b91004e3aba6b9117264b8f15b0b5e02679369f5700e9dae2fa4b"} Nov 03 00:08:34.707005 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:34.706579 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnv4gk" Nov 03 00:08:34.707005 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:34.706607 4304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30d643681a8b91004e3aba6b9117264b8f15b0b5e02679369f5700e9dae2fa4b" Nov 03 00:08:34.708533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:34.708506 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" event={"ID":"84b805c7-75f8-4230-ae03-5ef7f752c2f2","Type":"ContainerDied","Data":"1f64a86c6779e749fa8f92fd5fedbd66595174036575febabb876bd670d86665"} Nov 03 00:08:34.708659 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:34.708556 4304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f64a86c6779e749fa8f92fd5fedbd66595174036575febabb876bd670d86665" Nov 03 00:08:34.708659 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:34.708515 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5xnxr" Nov 03 00:08:40.556515 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556472 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5"] Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556896 4304 cpu_manager.go:414] "containerMap: removing container" podUID="65087784-5de2-45e7-bade-f484f33a41ea" containerName="util" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556914 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65087784-5de2-45e7-bade-f484f33a41ea" containerName="util" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556926 4304 cpu_manager.go:414] "containerMap: removing container" podUID="3a30d16c-4afd-4a8b-bba6-dc923bfea766" containerName="util" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556932 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a30d16c-4afd-4a8b-bba6-dc923bfea766" containerName="util" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556941 4304 cpu_manager.go:414] "containerMap: removing container" podUID="8b6b8f9a-9476-43e6-b84a-62db03481a4f" containerName="pull" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556948 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6b8f9a-9476-43e6-b84a-62db03481a4f" containerName="pull" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556963 4304 cpu_manager.go:414] "containerMap: removing container" podUID="65087784-5de2-45e7-bade-f484f33a41ea" containerName="pull" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556969 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65087784-5de2-45e7-bade-f484f33a41ea" containerName="pull" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556978 4304 cpu_manager.go:414] "containerMap: removing container" podUID="8b6b8f9a-9476-43e6-b84a-62db03481a4f" containerName="extract" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556984 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6b8f9a-9476-43e6-b84a-62db03481a4f" containerName="extract" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556993 4304 cpu_manager.go:414] "containerMap: removing container" podUID="65087784-5de2-45e7-bade-f484f33a41ea" containerName="extract" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.556999 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65087784-5de2-45e7-bade-f484f33a41ea" containerName="extract" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557007 4304 cpu_manager.go:414] "containerMap: removing container" podUID="84b805c7-75f8-4230-ae03-5ef7f752c2f2" containerName="extract" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557013 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b805c7-75f8-4230-ae03-5ef7f752c2f2" containerName="extract" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557028 4304 cpu_manager.go:414] "containerMap: removing container" podUID="84b805c7-75f8-4230-ae03-5ef7f752c2f2" containerName="util" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557035 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b805c7-75f8-4230-ae03-5ef7f752c2f2" containerName="util" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557043 4304 cpu_manager.go:414] "containerMap: removing container" podUID="e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" containerName="extract" Nov 03 00:08:40.557039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557049 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" containerName="extract" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557065 4304 cpu_manager.go:414] "containerMap: removing container" podUID="e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" containerName="pull" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557071 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" containerName="pull" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557082 4304 cpu_manager.go:414] "containerMap: removing container" podUID="3a30d16c-4afd-4a8b-bba6-dc923bfea766" containerName="extract" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557088 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a30d16c-4afd-4a8b-bba6-dc923bfea766" containerName="extract" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557100 4304 cpu_manager.go:414] "containerMap: removing container" podUID="3a30d16c-4afd-4a8b-bba6-dc923bfea766" containerName="pull" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557107 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a30d16c-4afd-4a8b-bba6-dc923bfea766" containerName="pull" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557117 4304 cpu_manager.go:414] "containerMap: removing container" podUID="8b6b8f9a-9476-43e6-b84a-62db03481a4f" containerName="util" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557123 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6b8f9a-9476-43e6-b84a-62db03481a4f" containerName="util" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557137 4304 cpu_manager.go:414] "containerMap: removing container" podUID="e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" containerName="util" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557144 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" containerName="util" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557155 4304 cpu_manager.go:414] "containerMap: removing container" podUID="84b805c7-75f8-4230-ae03-5ef7f752c2f2" containerName="pull" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557161 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b805c7-75f8-4230-ae03-5ef7f752c2f2" containerName="pull" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557221 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e504fb8b-9b06-4a3c-b557-d4eb6ecc8d7b" containerName="extract" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557234 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a30d16c-4afd-4a8b-bba6-dc923bfea766" containerName="extract" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557246 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6b8f9a-9476-43e6-b84a-62db03481a4f" containerName="extract" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557253 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="65087784-5de2-45e7-bade-f484f33a41ea" containerName="extract" Nov 03 00:08:40.557892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.557260 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b805c7-75f8-4230-ae03-5ef7f752c2f2" containerName="extract" Nov 03 00:08:40.585823 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.585794 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.623694 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.623664 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5"] Nov 03 00:08:40.645145 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.645106 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec08a22c-d101-4dc3-a8ce-46522a696a87-tmp\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.645295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.645227 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec08a22c-d101-4dc3-a8ce-46522a696a87-manifests\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.645295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.645279 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec08a22c-d101-4dc3-a8ce-46522a696a87-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.645419 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.645331 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec08a22c-d101-4dc3-a8ce-46522a696a87-metrics-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.645419 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.645354 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zsx\" (UniqueName: \"kubernetes.io/projected/ec08a22c-d101-4dc3-a8ce-46522a696a87-kube-api-access-28zsx\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.645419 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.645383 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec08a22c-d101-4dc3-a8ce-46522a696a87-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.679399 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.679374 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 00:08:40.679584 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.679568 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 00:08:40.679584 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.679577 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 00:08:40.679701 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.679585 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 00:08:40.679701 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.679579 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 00:08:40.679701 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.679634 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9xr6v\"" Nov 03 00:08:40.745738 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.745710 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec08a22c-d101-4dc3-a8ce-46522a696a87-manifests\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.745883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.745748 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec08a22c-d101-4dc3-a8ce-46522a696a87-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.745883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.745803 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec08a22c-d101-4dc3-a8ce-46522a696a87-metrics-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.745883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.745825 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zsx\" (UniqueName: \"kubernetes.io/projected/ec08a22c-d101-4dc3-a8ce-46522a696a87-kube-api-access-28zsx\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.745883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.745844 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec08a22c-d101-4dc3-a8ce-46522a696a87-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.745883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.745862 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec08a22c-d101-4dc3-a8ce-46522a696a87-tmp\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.746210 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.746169 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec08a22c-d101-4dc3-a8ce-46522a696a87-manifests\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.748505 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.748483 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec08a22c-d101-4dc3-a8ce-46522a696a87-tmp\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.748670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.748538 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec08a22c-d101-4dc3-a8ce-46522a696a87-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.748670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.748622 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec08a22c-d101-4dc3-a8ce-46522a696a87-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.748670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.748622 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec08a22c-d101-4dc3-a8ce-46522a696a87-metrics-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.804410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.804373 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zsx\" (UniqueName: \"kubernetes.io/projected/ec08a22c-d101-4dc3-a8ce-46522a696a87-kube-api-access-28zsx\") pod \"opendatahub-operator-controller-manager-65f97f758d-prgs5\" (UID: \"ec08a22c-d101-4dc3-a8ce-46522a696a87\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:40.929440 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:40.929415 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:41.413643 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:41.413620 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5"] Nov 03 00:08:41.416058 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:08:41.416023 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec08a22c_d101_4dc3_a8ce_46522a696a87.slice/crio-99602e23802e9669184636d4ba0cd70a508c86e6ea894f488170aeb8b5807ae3 WatchSource:0}: Error finding container 99602e23802e9669184636d4ba0cd70a508c86e6ea894f488170aeb8b5807ae3: Status 404 returned error can't find the container with id 99602e23802e9669184636d4ba0cd70a508c86e6ea894f488170aeb8b5807ae3 Nov 03 00:08:41.735242 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:41.735152 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" event={"ID":"ec08a22c-d101-4dc3-a8ce-46522a696a87","Type":"ContainerStarted","Data":"99602e23802e9669184636d4ba0cd70a508c86e6ea894f488170aeb8b5807ae3"} Nov 03 00:08:45.753238 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:45.753204 4304 generic.go:356] "Generic (PLEG): container finished" podID="ec08a22c-d101-4dc3-a8ce-46522a696a87" containerID="c8114a25076f0683b8a23ae37a0b5c44512902db9bf7f79a4615aa27c7634f97" exitCode=0 Nov 03 00:08:45.753712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:45.753251 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" event={"ID":"ec08a22c-d101-4dc3-a8ce-46522a696a87","Type":"ContainerDied","Data":"c8114a25076f0683b8a23ae37a0b5c44512902db9bf7f79a4615aa27c7634f97"} Nov 03 00:08:46.758826 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:46.758764 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" event={"ID":"ec08a22c-d101-4dc3-a8ce-46522a696a87","Type":"ContainerStarted","Data":"8c48bf41af93c3d49f551de72fd47cf7b2adf57770702cb97433c2c05437ec31"} Nov 03 00:08:46.759299 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:46.758916 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:46.988996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:46.988945 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" podStartSLOduration=2.443221147 podStartE2EDuration="6.988932731s" totalImagesPullingTime="4.099463529s" totalInitContainerRuntime="446.248055ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:46.938988733 +0000 UTC m=+210.222546407" watchObservedRunningTime="2026-11-03 00:08:46.988932731 +0000 UTC m=+210.272490404" Nov 03 00:08:48.168331 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.168298 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt"] Nov 03 00:08:48.170217 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.170202 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt" Nov 03 00:08:48.275835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.275643 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 00:08:48.275835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.275661 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 00:08:48.275835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.275720 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-llkq9\"" Nov 03 00:08:48.275835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.275725 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 00:08:48.277364 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.277341 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt"] Nov 03 00:08:48.309132 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.309105 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgrlg\" (UniqueName: \"kubernetes.io/projected/1c5959e1-69b6-46e0-94e3-3c6c2e0de70b-kube-api-access-kgrlg\") pod \"dns-operator-controller-manager-8557cbdb84-6bknt\" (UID: \"1c5959e1-69b6-46e0-94e3-3c6c2e0de70b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt" Nov 03 00:08:48.410060 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.410026 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgrlg\" (UniqueName: \"kubernetes.io/projected/1c5959e1-69b6-46e0-94e3-3c6c2e0de70b-kube-api-access-kgrlg\") pod \"dns-operator-controller-manager-8557cbdb84-6bknt\" (UID: \"1c5959e1-69b6-46e0-94e3-3c6c2e0de70b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt" Nov 03 00:08:48.467470 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.467394 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgrlg\" (UniqueName: \"kubernetes.io/projected/1c5959e1-69b6-46e0-94e3-3c6c2e0de70b-kube-api-access-kgrlg\") pod \"dns-operator-controller-manager-8557cbdb84-6bknt\" (UID: \"1c5959e1-69b6-46e0-94e3-3c6c2e0de70b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt" Nov 03 00:08:48.481070 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.481045 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt" Nov 03 00:08:48.855661 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:48.855610 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt"] Nov 03 00:08:48.858022 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:08:48.857995 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5959e1_69b6_46e0_94e3_3c6c2e0de70b.slice/crio-b4e4b843ceb965ebe2cb005a486aab60e64e40f5312baa18104f6ed57ac7fbd9 WatchSource:0}: Error finding container b4e4b843ceb965ebe2cb005a486aab60e64e40f5312baa18104f6ed57ac7fbd9: Status 404 returned error can't find the container with id b4e4b843ceb965ebe2cb005a486aab60e64e40f5312baa18104f6ed57ac7fbd9 Nov 03 00:08:49.770910 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:49.770875 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt" event={"ID":"1c5959e1-69b6-46e0-94e3-3c6c2e0de70b","Type":"ContainerStarted","Data":"b4e4b843ceb965ebe2cb005a486aab60e64e40f5312baa18104f6ed57ac7fbd9"} Nov 03 00:08:52.754380 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:52.754342 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b"] Nov 03 00:08:52.757835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:52.757819 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b" Nov 03 00:08:52.813333 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:52.813304 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-k9zft\"" Nov 03 00:08:52.827099 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:52.827070 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b"] Nov 03 00:08:52.850725 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:52.850694 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fzzd\" (UniqueName: \"kubernetes.io/projected/f9b55548-d7c6-4a1f-971b-0b679da43de1-kube-api-access-8fzzd\") pod \"authorino-operator-7b7bb4bc77-xlr9b\" (UID: \"f9b55548-d7c6-4a1f-971b-0b679da43de1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b" Nov 03 00:08:52.952117 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:52.952087 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fzzd\" (UniqueName: \"kubernetes.io/projected/f9b55548-d7c6-4a1f-971b-0b679da43de1-kube-api-access-8fzzd\") pod \"authorino-operator-7b7bb4bc77-xlr9b\" (UID: \"f9b55548-d7c6-4a1f-971b-0b679da43de1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b" Nov 03 00:08:53.012432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:53.012355 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fzzd\" (UniqueName: \"kubernetes.io/projected/f9b55548-d7c6-4a1f-971b-0b679da43de1-kube-api-access-8fzzd\") pod \"authorino-operator-7b7bb4bc77-xlr9b\" (UID: \"f9b55548-d7c6-4a1f-971b-0b679da43de1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b" Nov 03 00:08:53.068154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:53.068123 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b" Nov 03 00:08:53.446953 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:53.446923 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b"] Nov 03 00:08:53.482582 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:08:53.482556 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b55548_d7c6_4a1f_971b_0b679da43de1.slice/crio-1f91fe4463d0fa1d3617197b60f55d145e4efa57dcdb3217cbdb515756f2f561 WatchSource:0}: Error finding container 1f91fe4463d0fa1d3617197b60f55d145e4efa57dcdb3217cbdb515756f2f561: Status 404 returned error can't find the container with id 1f91fe4463d0fa1d3617197b60f55d145e4efa57dcdb3217cbdb515756f2f561 Nov 03 00:08:53.787510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:53.787426 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b" event={"ID":"f9b55548-d7c6-4a1f-971b-0b679da43de1","Type":"ContainerStarted","Data":"1f91fe4463d0fa1d3617197b60f55d145e4efa57dcdb3217cbdb515756f2f561"} Nov 03 00:08:57.764463 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:57.764433 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-prgs5" Nov 03 00:08:58.655455 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:08:58.655413 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:08:58.813369 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:58.813331 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b" event={"ID":"f9b55548-d7c6-4a1f-971b-0b679da43de1","Type":"ContainerStarted","Data":"ee274bbac5572508b85b842a085bd0fc37ab5bd5c3aeb09736daddaf45d16669"} Nov 03 00:08:58.813849 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:58.813444 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b" Nov 03 00:08:58.934310 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:08:58.934206 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b" podStartSLOduration=2.379155491 podStartE2EDuration="6.934190351s" totalImagesPullingTime="4.55503486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:58.932968446 +0000 UTC m=+222.216526119" watchObservedRunningTime="2026-11-03 00:08:58.934190351 +0000 UTC m=+222.217748027" Nov 03 00:09:01.825752 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:01.825653 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt" event={"ID":"1c5959e1-69b6-46e0-94e3-3c6c2e0de70b","Type":"ContainerStarted","Data":"25625591e7e80658d3c92ef801fa68314ef2d8b60a5972249dbdcb031ccf1f97"} Nov 03 00:09:01.826151 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:01.825754 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt" Nov 03 00:09:01.946071 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:01.946020 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt" podStartSLOduration=1.373090038 podStartE2EDuration="13.946008404s" totalImagesPullingTime="12.572918366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:01.94446892 +0000 UTC m=+225.228026593" watchObservedRunningTime="2026-11-03 00:09:01.946008404 +0000 UTC m=+225.229566078" Nov 03 00:09:02.352441 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.352393 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp"] Nov 03 00:09:02.355555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.355538 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp" Nov 03 00:09:02.415974 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.415951 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-pmzrn\"" Nov 03 00:09:02.421112 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.421088 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 00:09:02.429656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.429629 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efdb830e-196d-41d9-95e9-a48b32639570-tmp\") pod \"jobset-operator-674cd59dc5-bsmjp\" (UID: \"efdb830e-196d-41d9-95e9-a48b32639570\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp" Nov 03 00:09:02.429713 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.429679 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4c96\" (UniqueName: \"kubernetes.io/projected/efdb830e-196d-41d9-95e9-a48b32639570-kube-api-access-m4c96\") pod \"jobset-operator-674cd59dc5-bsmjp\" (UID: \"efdb830e-196d-41d9-95e9-a48b32639570\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp" Nov 03 00:09:02.459230 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.459199 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:09:02.460859 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.460838 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp"] Nov 03 00:09:02.530146 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.530115 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efdb830e-196d-41d9-95e9-a48b32639570-tmp\") pod \"jobset-operator-674cd59dc5-bsmjp\" (UID: \"efdb830e-196d-41d9-95e9-a48b32639570\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp" Nov 03 00:09:02.530301 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.530161 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4c96\" (UniqueName: \"kubernetes.io/projected/efdb830e-196d-41d9-95e9-a48b32639570-kube-api-access-m4c96\") pod \"jobset-operator-674cd59dc5-bsmjp\" (UID: \"efdb830e-196d-41d9-95e9-a48b32639570\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp" Nov 03 00:09:02.530505 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.530486 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efdb830e-196d-41d9-95e9-a48b32639570-tmp\") pod \"jobset-operator-674cd59dc5-bsmjp\" (UID: \"efdb830e-196d-41d9-95e9-a48b32639570\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp" Nov 03 00:09:02.587596 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.587560 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4c96\" (UniqueName: \"kubernetes.io/projected/efdb830e-196d-41d9-95e9-a48b32639570-kube-api-access-m4c96\") pod \"jobset-operator-674cd59dc5-bsmjp\" (UID: \"efdb830e-196d-41d9-95e9-a48b32639570\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp" Nov 03 00:09:02.679489 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:02.679454 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp" Nov 03 00:09:03.048480 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:03.048365 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp"] Nov 03 00:09:03.051097 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:09:03.051070 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefdb830e_196d_41d9_95e9_a48b32639570.slice/crio-4aac765a3160e49dc511996889f51598564c57cdda4e879ff529e8527634ca5d WatchSource:0}: Error finding container 4aac765a3160e49dc511996889f51598564c57cdda4e879ff529e8527634ca5d: Status 404 returned error can't find the container with id 4aac765a3160e49dc511996889f51598564c57cdda4e879ff529e8527634ca5d Nov 03 00:09:03.834420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:03.834382 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp" event={"ID":"efdb830e-196d-41d9-95e9-a48b32639570","Type":"ContainerStarted","Data":"4aac765a3160e49dc511996889f51598564c57cdda4e879ff529e8527634ca5d"} Nov 03 00:09:05.150262 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:05.150230 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb"] Nov 03 00:09:05.154698 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:05.154676 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb" Nov 03 00:09:05.209330 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:05.209301 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hn8f2\"" Nov 03 00:09:05.249608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:05.249576 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdjm7\" (UniqueName: \"kubernetes.io/projected/98df725b-22cd-4d9c-b61f-fcaae9c848dc-kube-api-access-zdjm7\") pod \"limitador-operator-controller-manager-5778bb4449-kg9lb\" (UID: \"98df725b-22cd-4d9c-b61f-fcaae9c848dc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb" Nov 03 00:09:05.260324 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:05.260298 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb"] Nov 03 00:09:05.350863 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:05.350832 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdjm7\" (UniqueName: \"kubernetes.io/projected/98df725b-22cd-4d9c-b61f-fcaae9c848dc-kube-api-access-zdjm7\") pod \"limitador-operator-controller-manager-5778bb4449-kg9lb\" (UID: \"98df725b-22cd-4d9c-b61f-fcaae9c848dc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb" Nov 03 00:09:05.416485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:05.416412 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdjm7\" (UniqueName: \"kubernetes.io/projected/98df725b-22cd-4d9c-b61f-fcaae9c848dc-kube-api-access-zdjm7\") pod \"limitador-operator-controller-manager-5778bb4449-kg9lb\" (UID: \"98df725b-22cd-4d9c-b61f-fcaae9c848dc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb" Nov 03 00:09:05.466534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:05.466507 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb" Nov 03 00:09:05.838832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:05.838682 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb"] Nov 03 00:09:05.841070 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:09:05.841035 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98df725b_22cd_4d9c_b61f_fcaae9c848dc.slice/crio-16d5a45a15632abb72d83ce389e95a4bb2f5147e12541b01be25e6de2d503342 WatchSource:0}: Error finding container 16d5a45a15632abb72d83ce389e95a4bb2f5147e12541b01be25e6de2d503342: Status 404 returned error can't find the container with id 16d5a45a15632abb72d83ce389e95a4bb2f5147e12541b01be25e6de2d503342 Nov 03 00:09:06.847424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:06.847386 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb" event={"ID":"98df725b-22cd-4d9c-b61f-fcaae9c848dc","Type":"ContainerStarted","Data":"16d5a45a15632abb72d83ce389e95a4bb2f5147e12541b01be25e6de2d503342"} Nov 03 00:09:08.858396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:08.858360 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp" event={"ID":"efdb830e-196d-41d9-95e9-a48b32639570","Type":"ContainerStarted","Data":"57cb33ea7d8d2507cf7d6461cded8a2e1704ec2e6c5161896ed75bfd2364dc50"} Nov 03 00:09:08.972884 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:08.972833 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bsmjp" podStartSLOduration=2.024412992 podStartE2EDuration="6.972819719s" totalImagesPullingTime="4.948406727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:08.971584327 +0000 UTC m=+232.255142004" watchObservedRunningTime="2026-11-03 00:09:08.972819719 +0000 UTC m=+232.256377392" Nov 03 00:09:09.818914 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:09.818884 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xlr9b" Nov 03 00:09:11.871737 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:11.871703 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb" event={"ID":"98df725b-22cd-4d9c-b61f-fcaae9c848dc","Type":"ContainerStarted","Data":"dd3f3e581f9f0f4124bc9e9075def2e77717e6fecdd9f2360085229b78e3c060"} Nov 03 00:09:11.872130 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:11.871820 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb" Nov 03 00:09:12.001669 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:12.001616 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb" podStartSLOduration=1.479393508 podStartE2EDuration="7.001602014s" totalImagesPullingTime="5.522208506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:12.000343771 +0000 UTC m=+235.283901464" watchObservedRunningTime="2026-11-03 00:09:12.001602014 +0000 UTC m=+235.285159688" Nov 03 00:09:12.831713 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:12.831685 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6bknt" Nov 03 00:09:17.597129 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.597098 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd"] Nov 03 00:09:17.662502 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.662468 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd"] Nov 03 00:09:17.662679 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.662581 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" Nov 03 00:09:17.715218 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.715193 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 00:09:17.715355 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.715302 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 00:09:17.751852 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.751821 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/71adc051-8d66-4db6-a823-2d912ff1d9ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-24fmd\" (UID: \"71adc051-8d66-4db6-a823-2d912ff1d9ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" Nov 03 00:09:17.751973 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.751862 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/71adc051-8d66-4db6-a823-2d912ff1d9ad-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-24fmd\" (UID: \"71adc051-8d66-4db6-a823-2d912ff1d9ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" Nov 03 00:09:17.751973 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.751879 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657jz\" (UniqueName: \"kubernetes.io/projected/71adc051-8d66-4db6-a823-2d912ff1d9ad-kube-api-access-657jz\") pod \"kuadrant-console-plugin-66c4fb575d-24fmd\" (UID: \"71adc051-8d66-4db6-a823-2d912ff1d9ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" Nov 03 00:09:17.764185 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.764165 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9fpwb\"" Nov 03 00:09:17.852461 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.852388 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/71adc051-8d66-4db6-a823-2d912ff1d9ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-24fmd\" (UID: \"71adc051-8d66-4db6-a823-2d912ff1d9ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" Nov 03 00:09:17.852461 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.852427 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/71adc051-8d66-4db6-a823-2d912ff1d9ad-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-24fmd\" (UID: \"71adc051-8d66-4db6-a823-2d912ff1d9ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" Nov 03 00:09:17.852461 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.852443 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657jz\" (UniqueName: \"kubernetes.io/projected/71adc051-8d66-4db6-a823-2d912ff1d9ad-kube-api-access-657jz\") pod \"kuadrant-console-plugin-66c4fb575d-24fmd\" (UID: \"71adc051-8d66-4db6-a823-2d912ff1d9ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" Nov 03 00:09:17.853121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.853102 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/71adc051-8d66-4db6-a823-2d912ff1d9ad-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-24fmd\" (UID: \"71adc051-8d66-4db6-a823-2d912ff1d9ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" Nov 03 00:09:17.855019 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.854998 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/71adc051-8d66-4db6-a823-2d912ff1d9ad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-24fmd\" (UID: \"71adc051-8d66-4db6-a823-2d912ff1d9ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" Nov 03 00:09:17.909574 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.909547 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657jz\" (UniqueName: \"kubernetes.io/projected/71adc051-8d66-4db6-a823-2d912ff1d9ad-kube-api-access-657jz\") pod \"kuadrant-console-plugin-66c4fb575d-24fmd\" (UID: \"71adc051-8d66-4db6-a823-2d912ff1d9ad\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" Nov 03 00:09:17.992478 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:17.992450 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" Nov 03 00:09:18.383516 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:18.383495 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd"] Nov 03 00:09:18.385826 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:09:18.385765 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71adc051_8d66_4db6_a823_2d912ff1d9ad.slice/crio-e93d3a936ced737ba44fd7689260998ed6c9911697e5f9bafda7e4ab1bc1dfe4 WatchSource:0}: Error finding container e93d3a936ced737ba44fd7689260998ed6c9911697e5f9bafda7e4ab1bc1dfe4: Status 404 returned error can't find the container with id e93d3a936ced737ba44fd7689260998ed6c9911697e5f9bafda7e4ab1bc1dfe4 Nov 03 00:09:18.899993 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:18.899959 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" event={"ID":"71adc051-8d66-4db6-a823-2d912ff1d9ad","Type":"ContainerStarted","Data":"e93d3a936ced737ba44fd7689260998ed6c9911697e5f9bafda7e4ab1bc1dfe4"} Nov 03 00:09:20.617680 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.617645 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54744779bc-xvccs"] Nov 03 00:09:20.619695 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.619671 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" Nov 03 00:09:20.671723 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.671695 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 00:09:20.671902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.671760 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 00:09:20.676487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.676242 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9q2\" (UniqueName: \"kubernetes.io/projected/78db11d2-e64f-431e-8460-f4838ff300bf-kube-api-access-lk9q2\") pod \"kube-auth-proxy-54744779bc-xvccs\" (UID: \"78db11d2-e64f-431e-8460-f4838ff300bf\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" Nov 03 00:09:20.676487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.676334 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78db11d2-e64f-431e-8460-f4838ff300bf-tmp\") pod \"kube-auth-proxy-54744779bc-xvccs\" (UID: \"78db11d2-e64f-431e-8460-f4838ff300bf\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" Nov 03 00:09:20.676487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.676383 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78db11d2-e64f-431e-8460-f4838ff300bf-tls-certs\") pod \"kube-auth-proxy-54744779bc-xvccs\" (UID: \"78db11d2-e64f-431e-8460-f4838ff300bf\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" Nov 03 00:09:20.711521 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.711444 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54744779bc-xvccs"] Nov 03 00:09:20.761794 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.761746 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 00:09:20.761794 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.761769 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-k9nfq\"" Nov 03 00:09:20.762003 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.761962 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 00:09:20.777464 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.777438 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9q2\" (UniqueName: \"kubernetes.io/projected/78db11d2-e64f-431e-8460-f4838ff300bf-kube-api-access-lk9q2\") pod \"kube-auth-proxy-54744779bc-xvccs\" (UID: \"78db11d2-e64f-431e-8460-f4838ff300bf\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" Nov 03 00:09:20.777579 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.777488 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78db11d2-e64f-431e-8460-f4838ff300bf-tmp\") pod \"kube-auth-proxy-54744779bc-xvccs\" (UID: \"78db11d2-e64f-431e-8460-f4838ff300bf\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" Nov 03 00:09:20.777579 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.777517 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78db11d2-e64f-431e-8460-f4838ff300bf-tls-certs\") pod \"kube-auth-proxy-54744779bc-xvccs\" (UID: \"78db11d2-e64f-431e-8460-f4838ff300bf\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" Nov 03 00:09:20.780000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.779976 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78db11d2-e64f-431e-8460-f4838ff300bf-tmp\") pod \"kube-auth-proxy-54744779bc-xvccs\" (UID: \"78db11d2-e64f-431e-8460-f4838ff300bf\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" Nov 03 00:09:20.780155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.780140 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78db11d2-e64f-431e-8460-f4838ff300bf-tls-certs\") pod \"kube-auth-proxy-54744779bc-xvccs\" (UID: \"78db11d2-e64f-431e-8460-f4838ff300bf\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" Nov 03 00:09:20.833715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.833693 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9q2\" (UniqueName: \"kubernetes.io/projected/78db11d2-e64f-431e-8460-f4838ff300bf-kube-api-access-lk9q2\") pod \"kube-auth-proxy-54744779bc-xvccs\" (UID: \"78db11d2-e64f-431e-8460-f4838ff300bf\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" Nov 03 00:09:20.934321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:20.934291 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" Nov 03 00:09:21.306450 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:21.306427 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54744779bc-xvccs"] Nov 03 00:09:21.309276 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:09:21.309243 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78db11d2_e64f_431e_8460_f4838ff300bf.slice/crio-23f6314cf3ed976546503a1197cfd35c6d72020c76c835313f140236b86afc81 WatchSource:0}: Error finding container 23f6314cf3ed976546503a1197cfd35c6d72020c76c835313f140236b86afc81: Status 404 returned error can't find the container with id 23f6314cf3ed976546503a1197cfd35c6d72020c76c835313f140236b86afc81 Nov 03 00:09:21.913873 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:21.913840 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" event={"ID":"78db11d2-e64f-431e-8460-f4838ff300bf","Type":"ContainerStarted","Data":"23f6314cf3ed976546503a1197cfd35c6d72020c76c835313f140236b86afc81"} Nov 03 00:09:22.878472 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:22.878442 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-kg9lb" Nov 03 00:09:25.935051 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:25.935008 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" event={"ID":"78db11d2-e64f-431e-8460-f4838ff300bf","Type":"ContainerStarted","Data":"35c14e8a71c5df21decaff77b40ff2e4fff53889ebffca536836e0c6f70f0022"} Nov 03 00:09:26.050136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:26.050072 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54744779bc-xvccs" podStartSLOduration=2.343959763 podStartE2EDuration="6.050053146s" totalImagesPullingTime="3.706093383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:26.049165715 +0000 UTC m=+249.332723390" watchObservedRunningTime="2026-11-03 00:09:26.050053146 +0000 UTC m=+249.333610820" Nov 03 00:09:27.946124 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:27.946087 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" event={"ID":"71adc051-8d66-4db6-a823-2d912ff1d9ad","Type":"ContainerStarted","Data":"4974f7a1ebf7221bdd1bc88d9cfff652c122ab280e45beac867c8685d5d6c259"} Nov 03 00:09:27.965917 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:27.965879 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-kdnft"] Nov 03 00:09:27.969290 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:27.969270 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-kdnft" Nov 03 00:09:28.044194 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.044166 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46a73aad-0ef9-4196-a80a-15927d9c0945-bound-sa-token\") pod \"cert-manager-597b95cb84-kdnft\" (UID: \"46a73aad-0ef9-4196-a80a-15927d9c0945\") " pod="cert-manager/cert-manager-597b95cb84-kdnft" Nov 03 00:09:28.044362 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.044210 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpcm8\" (UniqueName: \"kubernetes.io/projected/46a73aad-0ef9-4196-a80a-15927d9c0945-kube-api-access-tpcm8\") pod \"cert-manager-597b95cb84-kdnft\" (UID: \"46a73aad-0ef9-4196-a80a-15927d9c0945\") " pod="cert-manager/cert-manager-597b95cb84-kdnft" Nov 03 00:09:28.046912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.046891 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-444x6\"" Nov 03 00:09:28.047006 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.046926 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 00:09:28.047006 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.046993 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 00:09:28.048466 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.048447 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-kdnft"] Nov 03 00:09:28.074956 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.074918 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-24fmd" podStartSLOduration=2.141536462 podStartE2EDuration="11.074906236s" totalImagesPullingTime="8.933369774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:28.073454683 +0000 UTC m=+251.357012358" watchObservedRunningTime="2026-11-03 00:09:28.074906236 +0000 UTC m=+251.358463955" Nov 03 00:09:28.144869 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.144839 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpcm8\" (UniqueName: \"kubernetes.io/projected/46a73aad-0ef9-4196-a80a-15927d9c0945-kube-api-access-tpcm8\") pod \"cert-manager-597b95cb84-kdnft\" (UID: \"46a73aad-0ef9-4196-a80a-15927d9c0945\") " pod="cert-manager/cert-manager-597b95cb84-kdnft" Nov 03 00:09:28.144999 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.144905 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46a73aad-0ef9-4196-a80a-15927d9c0945-bound-sa-token\") pod \"cert-manager-597b95cb84-kdnft\" (UID: \"46a73aad-0ef9-4196-a80a-15927d9c0945\") " pod="cert-manager/cert-manager-597b95cb84-kdnft" Nov 03 00:09:28.200946 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.200880 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46a73aad-0ef9-4196-a80a-15927d9c0945-bound-sa-token\") pod \"cert-manager-597b95cb84-kdnft\" (UID: \"46a73aad-0ef9-4196-a80a-15927d9c0945\") " pod="cert-manager/cert-manager-597b95cb84-kdnft" Nov 03 00:09:28.200946 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.200889 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpcm8\" (UniqueName: \"kubernetes.io/projected/46a73aad-0ef9-4196-a80a-15927d9c0945-kube-api-access-tpcm8\") pod \"cert-manager-597b95cb84-kdnft\" (UID: \"46a73aad-0ef9-4196-a80a-15927d9c0945\") " pod="cert-manager/cert-manager-597b95cb84-kdnft" Nov 03 00:09:28.280800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.280735 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-kdnft" Nov 03 00:09:28.653914 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:09:28.653872 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:09:28.668943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.668919 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-kdnft"] Nov 03 00:09:28.670812 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:09:28.670765 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46a73aad_0ef9_4196_a80a_15927d9c0945.slice/crio-f97b22780ba26f449d11f195507e028f5bd1a2236589a4783ef4740add98da08 WatchSource:0}: Error finding container f97b22780ba26f449d11f195507e028f5bd1a2236589a4783ef4740add98da08: Status 404 returned error can't find the container with id f97b22780ba26f449d11f195507e028f5bd1a2236589a4783ef4740add98da08 Nov 03 00:09:28.951968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:28.951890 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-kdnft" event={"ID":"46a73aad-0ef9-4196-a80a-15927d9c0945","Type":"ContainerStarted","Data":"f97b22780ba26f449d11f195507e028f5bd1a2236589a4783ef4740add98da08"} Nov 03 00:09:32.632887 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.632853 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9"] Nov 03 00:09:32.635914 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.635885 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.687718 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.687684 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 00:09:32.687874 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.687758 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 00:09:32.737389 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.737352 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-d5688\"" Nov 03 00:09:32.786338 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.785989 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvq9\" (UniqueName: \"kubernetes.io/projected/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-kube-api-access-2mvq9\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.786338 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.786068 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.786338 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.786108 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.786338 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.786147 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.837011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.836980 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 00:09:32.838621 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.838600 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9"] Nov 03 00:09:32.887045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.886981 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.887045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.887023 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.887245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.887061 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvq9\" (UniqueName: \"kubernetes.io/projected/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-kube-api-access-2mvq9\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.887245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.887112 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.887452 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.887431 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.887534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.887512 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.887935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.887919 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.944063 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.944035 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvq9\" (UniqueName: \"kubernetes.io/projected/ff259ff5-f5a0-47f7-8a8e-f2f6125748e5-kube-api-access-2mvq9\") pod \"kserve-module-controller-manager-7566cf5fc8-rttv9\" (UID: \"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.949921 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.949885 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:32.972847 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:32.972811 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-kdnft" event={"ID":"46a73aad-0ef9-4196-a80a-15927d9c0945","Type":"ContainerStarted","Data":"04788cbfaf693284a4d5bff02f5c6ff53e45db58222159f5a79c01628a8a433e"} Nov 03 00:09:33.086265 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:33.086194 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-kdnft" podStartSLOduration=2.066537216 podStartE2EDuration="6.086176051s" totalImagesPullingTime="4.019638835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:33.08529313 +0000 UTC m=+256.368850804" watchObservedRunningTime="2026-11-03 00:09:33.086176051 +0000 UTC m=+256.369733726" Nov 03 00:09:33.366691 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:33.366663 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9"] Nov 03 00:09:33.369487 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:09:33.369459 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff259ff5_f5a0_47f7_8a8e_f2f6125748e5.slice/crio-ddbbef6e61eb596ccb1fb7b57fc6c33e865a4fee851e1c843478524f954a5e2b WatchSource:0}: Error finding container ddbbef6e61eb596ccb1fb7b57fc6c33e865a4fee851e1c843478524f954a5e2b: Status 404 returned error can't find the container with id ddbbef6e61eb596ccb1fb7b57fc6c33e865a4fee851e1c843478524f954a5e2b Nov 03 00:09:33.978558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:33.978517 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" event={"ID":"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5","Type":"ContainerStarted","Data":"ddbbef6e61eb596ccb1fb7b57fc6c33e865a4fee851e1c843478524f954a5e2b"} Nov 03 00:09:36.992633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:36.992535 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" event={"ID":"ff259ff5-f5a0-47f7-8a8e-f2f6125748e5","Type":"ContainerStarted","Data":"fda8b5c910614fe63c7041b5a4554f96ff66a59ee3b4c103ac504dbfaaf1159d"} Nov 03 00:09:36.993090 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:36.992683 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:37.156116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:37.156059 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" podStartSLOduration=1.860501897 podStartE2EDuration="5.156046167s" totalImagesPullingTime="3.29554427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:37.155694141 +0000 UTC m=+260.439251827" watchObservedRunningTime="2026-11-03 00:09:37.156046167 +0000 UTC m=+260.439603841" Nov 03 00:09:57.998447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:09:57.998419 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rttv9" Nov 03 00:09:58.654050 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:09:58.654008 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:10:28.657880 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:10:28.657842 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:10:39.367916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.367840 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-dzrml"] Nov 03 00:10:39.373468 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.373442 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:39.476002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.475968 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kcqvb\"" Nov 03 00:10:39.476176 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.476029 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 00:10:39.476176 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.476057 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 00:10:39.477784 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.477746 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dzrml"] Nov 03 00:10:39.518748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.518719 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6vl\" (UniqueName: \"kubernetes.io/projected/22dc8845-4b7d-41fb-9c91-cebde8326e19-kube-api-access-sc6vl\") pod \"odh-model-controller-769875db56-dzrml\" (UID: \"22dc8845-4b7d-41fb-9c91-cebde8326e19\") " pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:39.518899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.518755 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dc8845-4b7d-41fb-9c91-cebde8326e19-metrics-certs\") pod \"odh-model-controller-769875db56-dzrml\" (UID: \"22dc8845-4b7d-41fb-9c91-cebde8326e19\") " pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:39.518899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.518889 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22dc8845-4b7d-41fb-9c91-cebde8326e19-cert\") pod \"odh-model-controller-769875db56-dzrml\" (UID: \"22dc8845-4b7d-41fb-9c91-cebde8326e19\") " pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:39.620167 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.620093 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6vl\" (UniqueName: \"kubernetes.io/projected/22dc8845-4b7d-41fb-9c91-cebde8326e19-kube-api-access-sc6vl\") pod \"odh-model-controller-769875db56-dzrml\" (UID: \"22dc8845-4b7d-41fb-9c91-cebde8326e19\") " pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:39.620167 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.620133 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dc8845-4b7d-41fb-9c91-cebde8326e19-metrics-certs\") pod \"odh-model-controller-769875db56-dzrml\" (UID: \"22dc8845-4b7d-41fb-9c91-cebde8326e19\") " pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:39.620331 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.620251 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22dc8845-4b7d-41fb-9c91-cebde8326e19-cert\") pod \"odh-model-controller-769875db56-dzrml\" (UID: \"22dc8845-4b7d-41fb-9c91-cebde8326e19\") " pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:39.622810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.622768 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22dc8845-4b7d-41fb-9c91-cebde8326e19-cert\") pod \"odh-model-controller-769875db56-dzrml\" (UID: \"22dc8845-4b7d-41fb-9c91-cebde8326e19\") " pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:39.622923 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.622831 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dc8845-4b7d-41fb-9c91-cebde8326e19-metrics-certs\") pod \"odh-model-controller-769875db56-dzrml\" (UID: \"22dc8845-4b7d-41fb-9c91-cebde8326e19\") " pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:39.676712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.676684 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6vl\" (UniqueName: \"kubernetes.io/projected/22dc8845-4b7d-41fb-9c91-cebde8326e19-kube-api-access-sc6vl\") pod \"odh-model-controller-769875db56-dzrml\" (UID: \"22dc8845-4b7d-41fb-9c91-cebde8326e19\") " pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:39.689550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:39.689531 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:40.056340 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:40.056317 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dzrml"] Nov 03 00:10:40.058298 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:10:40.058267 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dc8845_4b7d_41fb_9c91_cebde8326e19.slice/crio-467a1b8b4323892eb04ef927dc1d0942b4cc198f66af65b3962de00a0bbd2896 WatchSource:0}: Error finding container 467a1b8b4323892eb04ef927dc1d0942b4cc198f66af65b3962de00a0bbd2896: Status 404 returned error can't find the container with id 467a1b8b4323892eb04ef927dc1d0942b4cc198f66af65b3962de00a0bbd2896 Nov 03 00:10:40.059553 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:40.059537 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:10:40.266078 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:40.266040 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dzrml" event={"ID":"22dc8845-4b7d-41fb-9c91-cebde8326e19","Type":"ContainerStarted","Data":"467a1b8b4323892eb04ef927dc1d0942b4cc198f66af65b3962de00a0bbd2896"} Nov 03 00:10:44.290828 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:44.290723 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dzrml" event={"ID":"22dc8845-4b7d-41fb-9c91-cebde8326e19","Type":"ContainerStarted","Data":"966d87b7c8b76266b97dfef9a13d31623c7172070172617df81f8db6041cab7c"} Nov 03 00:10:44.291201 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:44.290923 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:44.404015 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:44.403962 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-dzrml" podStartSLOduration=1.488831255 podStartE2EDuration="5.403945523s" totalImagesPullingTime="3.915114268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:44.403045382 +0000 UTC m=+327.686603057" watchObservedRunningTime="2026-11-03 00:10:44.403945523 +0000 UTC m=+327.687503219" Nov 03 00:10:55.297737 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:10:55.297702 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-dzrml" Nov 03 00:10:58.654478 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:10:58.654440 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:11:03.578888 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.578851 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2"] Nov 03 00:11:03.581948 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.581929 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.621708 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.621677 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-metrics-certs\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.621708 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.621710 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-maas-api-tls\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.621930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.621733 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9q4\" (UniqueName: \"kubernetes.io/projected/c158dc72-e23a-4f8d-9b7d-53ef9f347281-kube-api-access-ps9q4\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.621930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.621821 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c158dc72-e23a-4f8d-9b7d-53ef9f347281-openshift-service-ca\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.683408 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.683381 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:11:03.683566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.683411 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:11:03.683566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.683446 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kgpd5\"" Nov 03 00:11:03.683646 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.683638 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:11:03.683817 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.683799 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 00:11:03.685227 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.685210 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2"] Nov 03 00:11:03.723071 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.723041 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c158dc72-e23a-4f8d-9b7d-53ef9f347281-openshift-service-ca\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.723209 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.723097 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-metrics-certs\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.723209 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.723118 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-maas-api-tls\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.723209 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.723154 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9q4\" (UniqueName: \"kubernetes.io/projected/c158dc72-e23a-4f8d-9b7d-53ef9f347281-kube-api-access-ps9q4\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.723796 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.723754 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c158dc72-e23a-4f8d-9b7d-53ef9f347281-openshift-service-ca\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.725623 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.725604 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-maas-api-tls\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.725710 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.725657 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-metrics-certs\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.779069 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.779038 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9q4\" (UniqueName: \"kubernetes.io/projected/c158dc72-e23a-4f8d-9b7d-53ef9f347281-kube-api-access-ps9q4\") pod \"maas-api-d8998cc8c-hdmr2\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:03.893170 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:03.893141 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:04.260301 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:04.260274 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2"] Nov 03 00:11:04.261814 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:11:04.261788 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc158dc72_e23a_4f8d_9b7d_53ef9f347281.slice/crio-ae6e395514e02163339dc8950e1bd8a1da51bb64e9370d442155cbbcce6c8f19 WatchSource:0}: Error finding container ae6e395514e02163339dc8950e1bd8a1da51bb64e9370d442155cbbcce6c8f19: Status 404 returned error can't find the container with id ae6e395514e02163339dc8950e1bd8a1da51bb64e9370d442155cbbcce6c8f19 Nov 03 00:11:04.378938 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:04.378906 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" event={"ID":"c158dc72-e23a-4f8d-9b7d-53ef9f347281","Type":"ContainerStarted","Data":"ae6e395514e02163339dc8950e1bd8a1da51bb64e9370d442155cbbcce6c8f19"} Nov 03 00:11:07.396050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:07.396010 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" event={"ID":"c158dc72-e23a-4f8d-9b7d-53ef9f347281","Type":"ContainerStarted","Data":"e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702"} Nov 03 00:11:07.396420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:07.396147 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:07.510955 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:07.510901 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" podStartSLOduration=1.630629523 podStartE2EDuration="4.510886211s" totalImagesPullingTime="2.880256688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:07.509582833 +0000 UTC m=+350.793140508" watchObservedRunningTime="2026-11-03 00:11:07.510886211 +0000 UTC m=+350.794443885" Nov 03 00:11:13.405403 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:11:13.405372 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 00:11:28.653765 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:11:28.653727 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:11:58.653827 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:11:58.653791 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:12:28.654357 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:12:28.654321 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:12:58.657200 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:12:58.654635 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:13:01.339315 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.339282 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw"] Nov 03 00:13:01.342406 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.342390 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.444473 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.444419 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 00:13:01.444473 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.444419 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ggwn7\"" Nov 03 00:13:01.444473 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.444434 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 00:13:01.444737 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.444520 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 00:13:01.446155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.446135 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw"] Nov 03 00:13:01.473744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.473704 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.473934 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.473760 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.473934 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.473811 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.474100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.473983 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.474100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.474020 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg8j8\" (UniqueName: \"kubernetes.io/projected/00aa9eb4-4d67-4d73-9660-d7a3469465db-kube-api-access-gg8j8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.474239 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.474215 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00aa9eb4-4d67-4d73-9660-d7a3469465db-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.574957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.574921 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.574957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.574955 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.575208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.574980 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.575208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.575111 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg8j8\" (UniqueName: \"kubernetes.io/projected/00aa9eb4-4d67-4d73-9660-d7a3469465db-kube-api-access-gg8j8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.575208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.575180 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00aa9eb4-4d67-4d73-9660-d7a3469465db-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.575367 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.575227 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.575539 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.575515 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.575614 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.575560 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.575614 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.575576 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.577420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.577400 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00aa9eb4-4d67-4d73-9660-d7a3469465db-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.577672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.577656 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00aa9eb4-4d67-4d73-9660-d7a3469465db-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.632022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.631941 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg8j8\" (UniqueName: \"kubernetes.io/projected/00aa9eb4-4d67-4d73-9660-d7a3469465db-kube-api-access-gg8j8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw\" (UID: \"00aa9eb4-4d67-4d73-9660-d7a3469465db\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:01.654847 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:01.654817 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:02.023463 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:02.023437 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw"] Nov 03 00:13:02.025794 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:13:02.025750 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00aa9eb4_4d67_4d73_9660_d7a3469465db.slice/crio-4947729ffc3c070fad426d2f7f2b16c2b7875cfd24aaef2fb8a4798d148b805e WatchSource:0}: Error finding container 4947729ffc3c070fad426d2f7f2b16c2b7875cfd24aaef2fb8a4798d148b805e: Status 404 returned error can't find the container with id 4947729ffc3c070fad426d2f7f2b16c2b7875cfd24aaef2fb8a4798d148b805e Nov 03 00:13:02.902758 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:02.902714 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" event={"ID":"00aa9eb4-4d67-4d73-9660-d7a3469465db","Type":"ContainerStarted","Data":"4947729ffc3c070fad426d2f7f2b16c2b7875cfd24aaef2fb8a4798d148b805e"} Nov 03 00:13:04.914480 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:04.914445 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" event={"ID":"00aa9eb4-4d67-4d73-9660-d7a3469465db","Type":"ContainerStarted","Data":"e68274508287cc8e1714b967701f87a9393878ed2e2775ae2fed6ddaee3df96b"} Nov 03 00:13:05.029126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:05.029070 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" podStartSLOduration=1.419884733 podStartE2EDuration="4.029056495s" totalImagesPullingTime="2.609171762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:05.027049567 +0000 UTC m=+468.310607241" watchObservedRunningTime="2026-11-03 00:13:05.029056495 +0000 UTC m=+468.312614168" Nov 03 00:13:05.655507 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:05.655475 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:07.867952 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:07.867914 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv"] Nov 03 00:13:07.871494 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:07.871472 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:07.934944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:07.934913 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:07.934944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:07.934945 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:07.935211 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:07.935037 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:07.935211 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:07.935077 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh8nd\" (UniqueName: \"kubernetes.io/projected/7d27c868-cfaa-4629-a287-390559390fc1-kube-api-access-xh8nd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:07.935211 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:07.935100 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d27c868-cfaa-4629-a287-390559390fc1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:07.935211 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:07.935127 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:07.972323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:07.972295 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 00:13:07.974249 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:07.974230 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv"] Nov 03 00:13:08.036290 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.036242 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.036448 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.036339 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.036448 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.036368 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.036448 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.036423 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.036584 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.036470 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh8nd\" (UniqueName: \"kubernetes.io/projected/7d27c868-cfaa-4629-a287-390559390fc1-kube-api-access-xh8nd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.036656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.036631 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.036716 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.036673 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d27c868-cfaa-4629-a287-390559390fc1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.036832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.036810 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.036948 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.036898 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.038814 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.038764 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d27c868-cfaa-4629-a287-390559390fc1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.039307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.039289 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d27c868-cfaa-4629-a287-390559390fc1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.093471 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.093439 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh8nd\" (UniqueName: \"kubernetes.io/projected/7d27c868-cfaa-4629-a287-390559390fc1-kube-api-access-xh8nd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-h5npv\" (UID: \"7d27c868-cfaa-4629-a287-390559390fc1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.183728 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.183692 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:08.551032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.551009 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv"] Nov 03 00:13:08.553116 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:13:08.553090 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d27c868_cfaa_4629_a287_390559390fc1.slice/crio-362a7c9b473cdc4dfd24c23360a82784405391fc3541c9c909d49e3cbf3c20e4 WatchSource:0}: Error finding container 362a7c9b473cdc4dfd24c23360a82784405391fc3541c9c909d49e3cbf3c20e4: Status 404 returned error can't find the container with id 362a7c9b473cdc4dfd24c23360a82784405391fc3541c9c909d49e3cbf3c20e4 Nov 03 00:13:08.933653 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:08.933621 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" event={"ID":"7d27c868-cfaa-4629-a287-390559390fc1","Type":"ContainerStarted","Data":"362a7c9b473cdc4dfd24c23360a82784405391fc3541c9c909d49e3cbf3c20e4"} Nov 03 00:13:09.940222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:09.940185 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" event={"ID":"7d27c868-cfaa-4629-a287-390559390fc1","Type":"ContainerStarted","Data":"3a2cfc32676f54b1112cf47f1da6ba624247b6ea84569c66ebc0d6d1e355eec1"} Nov 03 00:13:10.054880 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:10.054828 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" podStartSLOduration=2.446987625 podStartE2EDuration="3.054814074s" totalImagesPullingTime="607.826449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:10.052594696 +0000 UTC m=+473.336152369" watchObservedRunningTime="2026-11-03 00:13:10.054814074 +0000 UTC m=+473.338371748" Nov 03 00:13:10.183980 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:10.183943 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:11.655468 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:11.655421 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:11.667991 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:11.667966 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:11.960569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:11.960491 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw" Nov 03 00:13:18.183859 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:18.183821 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:18.196260 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:18.196236 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:18.990443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:18.990413 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-h5npv" Nov 03 00:13:19.155700 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.155670 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm"] Nov 03 00:13:19.159507 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.159486 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.211634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.211605 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 00:13:19.218985 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.218951 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm"] Nov 03 00:13:19.231045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.231016 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71b643c-e306-4416-bd21-293c67b68b1e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.231154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.231074 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.231204 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.231179 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.231245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.231217 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.231245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.231236 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.231323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.231303 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7j7\" (UniqueName: \"kubernetes.io/projected/b71b643c-e306-4416-bd21-293c67b68b1e-kube-api-access-jv7j7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.332240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.332161 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.332240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.332210 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.332455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.332328 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.332455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.332357 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.332455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.332378 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7j7\" (UniqueName: \"kubernetes.io/projected/b71b643c-e306-4416-bd21-293c67b68b1e-kube-api-access-jv7j7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.332455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.332426 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71b643c-e306-4416-bd21-293c67b68b1e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.332648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.332550 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.332648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.332620 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.332714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.332683 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.334560 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.334535 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71b643c-e306-4416-bd21-293c67b68b1e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.334932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.334914 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71b643c-e306-4416-bd21-293c67b68b1e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.388387 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.388357 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7j7\" (UniqueName: \"kubernetes.io/projected/b71b643c-e306-4416-bd21-293c67b68b1e-kube-api-access-jv7j7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm\" (UID: \"b71b643c-e306-4416-bd21-293c67b68b1e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.471489 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.471463 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:19.839510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.839483 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm"] Nov 03 00:13:19.841565 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:13:19.841539 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71b643c_e306_4416_bd21_293c67b68b1e.slice/crio-44a57acfcde05e99a396002d28ac075f884f080bbc37d846afee39d26dc9f560 WatchSource:0}: Error finding container 44a57acfcde05e99a396002d28ac075f884f080bbc37d846afee39d26dc9f560: Status 404 returned error can't find the container with id 44a57acfcde05e99a396002d28ac075f884f080bbc37d846afee39d26dc9f560 Nov 03 00:13:19.985661 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:19.985628 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" event={"ID":"b71b643c-e306-4416-bd21-293c67b68b1e","Type":"ContainerStarted","Data":"44a57acfcde05e99a396002d28ac075f884f080bbc37d846afee39d26dc9f560"} Nov 03 00:13:20.992190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:20.992148 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" event={"ID":"b71b643c-e306-4416-bd21-293c67b68b1e","Type":"ContainerStarted","Data":"c7c458c327058c7dde4aa77f02a99bfd41e704cd083a5d4134b87151624dae81"} Nov 03 00:13:21.105466 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:21.105410 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" podStartSLOduration=1.521144684 podStartE2EDuration="2.105393114s" totalImagesPullingTime="584.24843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:21.104003613 +0000 UTC m=+484.387561287" watchObservedRunningTime="2026-11-03 00:13:21.105393114 +0000 UTC m=+484.388950821" Nov 03 00:13:21.471595 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:21.471562 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:28.654264 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:13:28.654225 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:13:29.471983 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:29.471912 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:29.484422 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:29.484386 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:30.043743 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.043713 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm" Nov 03 00:13:30.187298 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.187265 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6"] Nov 03 00:13:30.191341 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.191322 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.243103 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.243071 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 00:13:30.272820 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.272790 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6"] Nov 03 00:13:30.325739 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.325664 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.325739 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.325706 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.325739 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.325734 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.325958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.325860 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.325958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.325907 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftz2\" (UniqueName: \"kubernetes.io/projected/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-kube-api-access-bftz2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.326032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.326010 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.427327 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.427297 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.427327 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.427332 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.427558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.427353 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.427558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.427394 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftz2\" (UniqueName: \"kubernetes.io/projected/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-kube-api-access-bftz2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.427558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.427459 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.427558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.427502 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.427802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.427714 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.427867 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.427819 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.427905 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.427863 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.429730 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.429703 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.429908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.429892 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.484239 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.484206 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftz2\" (UniqueName: \"kubernetes.io/projected/49aef8b0-ab85-4ccc-bb3d-c5eebfa27469-kube-api-access-bftz2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6\" (UID: \"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.504888 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.504861 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:30.873413 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:30.873391 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6"] Nov 03 00:13:30.875652 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:13:30.875626 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49aef8b0_ab85_4ccc_bb3d_c5eebfa27469.slice/crio-e322004790c62a6ca13f9846f20289e73d562ea7831d171b57fb97507eaa418a WatchSource:0}: Error finding container e322004790c62a6ca13f9846f20289e73d562ea7831d171b57fb97507eaa418a: Status 404 returned error can't find the container with id e322004790c62a6ca13f9846f20289e73d562ea7831d171b57fb97507eaa418a Nov 03 00:13:31.039540 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:31.039505 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" event={"ID":"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469","Type":"ContainerStarted","Data":"e322004790c62a6ca13f9846f20289e73d562ea7831d171b57fb97507eaa418a"} Nov 03 00:13:32.046618 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:32.046580 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" event={"ID":"49aef8b0-ab85-4ccc-bb3d-c5eebfa27469","Type":"ContainerStarted","Data":"4532078ee3d1cf904aa5965ff1c73c55a8b1d7fcce03f6b154294ddf55c60abd"} Nov 03 00:13:32.163388 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:32.163331 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" podStartSLOduration=1.573097236 podStartE2EDuration="2.163317551s" totalImagesPullingTime="590.220315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:32.16084433 +0000 UTC m=+495.444402000" watchObservedRunningTime="2026-11-03 00:13:32.163317551 +0000 UTC m=+495.446875225" Nov 03 00:13:32.505314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:32.505282 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:34.235040 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.235006 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76"] Nov 03 00:13:34.238081 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.238062 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.290330 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.290298 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 00:13:34.339762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.339731 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zx4cj\"" Nov 03 00:13:34.342379 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.342350 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76"] Nov 03 00:13:34.361986 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.361957 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.362142 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.362007 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.362142 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.362025 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.362142 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.362082 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.362303 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.362232 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.362303 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.362275 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvkw6\" (UniqueName: \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-kube-api-access-fvkw6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.362404 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.362327 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.362404 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.362354 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.362499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.362408 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.462985 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.462948 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.462985 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.462985 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.463216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.463009 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.463216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.463040 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.463216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.463057 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.463216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.463074 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.463216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.463116 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.463216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.463158 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvkw6\" (UniqueName: \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-kube-api-access-fvkw6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.463216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.463186 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.463566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.463538 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.463737 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.463563 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.463737 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.463710 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.463869 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.463663 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.464092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.464073 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.466025 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.466003 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.466156 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.466141 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.520007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.519940 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.520221 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.520187 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvkw6\" (UniqueName: \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-kube-api-access-fvkw6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6wc76\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.550555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.550528 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:34.973994 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:34.973968 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76"] Nov 03 00:13:34.975709 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:13:34.975681 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8636cf_f4f7_440d_86a4_fb4dc5aa771c.slice/crio-1a6fbf0ae667ccbd345da4c42bba024bb8d055485adbbefa2156fd1c8e6da8e0 WatchSource:0}: Error finding container 1a6fbf0ae667ccbd345da4c42bba024bb8d055485adbbefa2156fd1c8e6da8e0: Status 404 returned error can't find the container with id 1a6fbf0ae667ccbd345da4c42bba024bb8d055485adbbefa2156fd1c8e6da8e0 Nov 03 00:13:35.060159 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:35.060130 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" event={"ID":"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c","Type":"ContainerStarted","Data":"1a6fbf0ae667ccbd345da4c42bba024bb8d055485adbbefa2156fd1c8e6da8e0"} Nov 03 00:13:40.435626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:40.435595 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:13:40.435952 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:40.435662 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:13:40.435952 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:40.435693 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:13:40.505878 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:40.505847 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:40.522806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:40.522757 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:41.091325 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:41.091288 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" event={"ID":"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c","Type":"ContainerStarted","Data":"df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b"} Nov 03 00:13:41.102471 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:41.102446 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:13:41.212147 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:41.212097 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" podStartSLOduration=1.7550231790000002 podStartE2EDuration="7.21208188s" totalImagesPullingTime="5.457058701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:41.211243318 +0000 UTC m=+504.494800993" watchObservedRunningTime="2026-11-03 00:13:41.21208188 +0000 UTC m=+504.495639615" Nov 03 00:13:41.551111 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:41.551072 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:41.552695 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:41.552673 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:13:41.552759 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:41.552721 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 00:13:42.551293 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:42.551257 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:13:42.551663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:42.551323 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 00:13:43.551694 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:43.551656 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:13:43.552099 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:43.551712 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 00:13:44.551723 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:44.551679 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:13:44.552141 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:44.551741 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 00:13:45.551088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:45.551043 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:13:45.551245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:45.551115 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 00:13:46.551153 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:46.551107 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:13:46.551655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:46.551166 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 00:13:47.551278 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:47.551235 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 00:13:47.551675 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:47.551304 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 00:13:48.652529 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:48.652498 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:48.652997 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:48.652873 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:48.653703 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:13:48.653683 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:13:58.653658 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:13:58.653622 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:14:28.655009 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:14:28.654964 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:14:42.828824 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.828794 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz"] Nov 03 00:14:42.832026 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.832005 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:42.883667 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.883636 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-168f474d-openshift-default-dockercfg-sjt2z\"" Nov 03 00:14:42.906206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.906176 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-data\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:42.906206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.906206 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbh4j\" (UniqueName: \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-kube-api-access-rbh4j\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:42.906395 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.906226 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-credential-socket\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:42.906395 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.906304 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-envoy\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:42.906395 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.906391 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istiod-ca-cert\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:42.906536 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.906502 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-socket\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:42.906595 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.906545 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-certs\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:42.906707 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.906677 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-podinfo\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:42.906827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.906726 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-token\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:42.936174 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:42.936148 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz"] Nov 03 00:14:43.007652 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.007616 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istiod-ca-cert\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.007864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.007658 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-socket\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.007864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.007680 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-certs\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.007864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.007696 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-podinfo\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.007864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.007715 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-token\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.007864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.007803 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-data\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.007864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.007828 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbh4j\" (UniqueName: \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-kube-api-access-rbh4j\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.007864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.007854 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-credential-socket\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.008361 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.007895 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-envoy\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.008621 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.008590 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-certs\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.008989 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.008954 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-socket\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.009102 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.009019 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-data\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.009240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.009219 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-credential-socket\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.009350 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.009330 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istiod-ca-cert\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.010787 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.010750 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-podinfo\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.011076 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.011056 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-envoy\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.064236 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.064206 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-token\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.064623 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.064603 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbh4j\" (UniqueName: \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-kube-api-access-rbh4j\") pod \"e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.096488 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.096422 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv"] Nov 03 00:14:43.100748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.100733 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:14:43.143938 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.143908 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:43.158059 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.157938 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv"] Nov 03 00:14:43.159729 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.159338 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2ccce380\"" Nov 03 00:14:43.198743 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.198524 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2ccce380-dockercfg-gf4v7\"" Nov 03 00:14:43.210272 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.210237 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qpn9\" (UniqueName: \"kubernetes.io/projected/289daeb7-1acb-4aad-a229-af361e633761-kube-api-access-7qpn9\") pod \"payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv\" (UID: \"289daeb7-1acb-4aad-a229-af361e633761\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:14:43.210378 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.210329 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289daeb7-1acb-4aad-a229-af361e633761-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv\" (UID: \"289daeb7-1acb-4aad-a229-af361e633761\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:14:43.230832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.230637 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7"] Nov 03 00:14:43.234820 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.234800 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:14:43.293589 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.293561 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7"] Nov 03 00:14:43.311555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.311529 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qpn9\" (UniqueName: \"kubernetes.io/projected/289daeb7-1acb-4aad-a229-af361e633761-kube-api-access-7qpn9\") pod \"payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv\" (UID: \"289daeb7-1acb-4aad-a229-af361e633761\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:14:43.311733 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.311574 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade6cd92-b99f-4799-b73d-878bece393d9-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7\" (UID: \"ade6cd92-b99f-4799-b73d-878bece393d9\") " pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:14:43.311733 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.311625 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blpr8\" (UniqueName: \"kubernetes.io/projected/ade6cd92-b99f-4799-b73d-878bece393d9-kube-api-access-blpr8\") pod \"payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7\" (UID: \"ade6cd92-b99f-4799-b73d-878bece393d9\") " pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:14:43.311733 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.311675 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289daeb7-1acb-4aad-a229-af361e633761-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv\" (UID: \"289daeb7-1acb-4aad-a229-af361e633761\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:14:43.312362 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.312339 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289daeb7-1acb-4aad-a229-af361e633761-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv\" (UID: \"289daeb7-1acb-4aad-a229-af361e633761\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:14:43.367834 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.367745 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qpn9\" (UniqueName: \"kubernetes.io/projected/289daeb7-1acb-4aad-a229-af361e633761-kube-api-access-7qpn9\") pod \"payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv\" (UID: \"289daeb7-1acb-4aad-a229-af361e633761\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:14:43.412422 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.412395 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade6cd92-b99f-4799-b73d-878bece393d9-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7\" (UID: \"ade6cd92-b99f-4799-b73d-878bece393d9\") " pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:14:43.412548 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.412441 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blpr8\" (UniqueName: \"kubernetes.io/projected/ade6cd92-b99f-4799-b73d-878bece393d9-kube-api-access-blpr8\") pod \"payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7\" (UID: \"ade6cd92-b99f-4799-b73d-878bece393d9\") " pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:14:43.413144 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.413122 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade6cd92-b99f-4799-b73d-878bece393d9-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7\" (UID: \"ade6cd92-b99f-4799-b73d-878bece393d9\") " pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:14:43.414320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.414300 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:14:43.469570 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.469544 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blpr8\" (UniqueName: \"kubernetes.io/projected/ade6cd92-b99f-4799-b73d-878bece393d9-kube-api-access-blpr8\") pod \"payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7\" (UID: \"ade6cd92-b99f-4799-b73d-878bece393d9\") " pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:14:43.550190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.550155 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:14:43.569533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.569508 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz"] Nov 03 00:14:43.571479 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:14:43.571453 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135c8d1b_e9c2_4f5a_8b05_ea277767f83b.slice/crio-21fbed81b8a35c773ee079b7cb566cf754ecd5cfbff676da915112578b32985f WatchSource:0}: Error finding container 21fbed81b8a35c773ee079b7cb566cf754ecd5cfbff676da915112578b32985f: Status 404 returned error can't find the container with id 21fbed81b8a35c773ee079b7cb566cf754ecd5cfbff676da915112578b32985f Nov 03 00:14:43.573734 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.573704 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:43.573870 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.573823 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:43.573870 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.573861 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:43.779710 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.779672 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv"] Nov 03 00:14:43.781713 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:14:43.781686 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289daeb7_1acb_4aad_a229_af361e633761.slice/crio-8e46e7992b78f6e5f16a7b9552ef2358def0c30f7a1d0ac1a1f4510a806a74fc WatchSource:0}: Error finding container 8e46e7992b78f6e5f16a7b9552ef2358def0c30f7a1d0ac1a1f4510a806a74fc: Status 404 returned error can't find the container with id 8e46e7992b78f6e5f16a7b9552ef2358def0c30f7a1d0ac1a1f4510a806a74fc Nov 03 00:14:43.920784 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:43.920740 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7"] Nov 03 00:14:43.922792 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:14:43.922747 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade6cd92_b99f_4799_b73d_878bece393d9.slice/crio-ca67ed23f49886b9bcf665be259dfb67db5e64dba70e6bcb81f1ed6474871082 WatchSource:0}: Error finding container ca67ed23f49886b9bcf665be259dfb67db5e64dba70e6bcb81f1ed6474871082: Status 404 returned error can't find the container with id ca67ed23f49886b9bcf665be259dfb67db5e64dba70e6bcb81f1ed6474871082 Nov 03 00:14:44.385172 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:44.385093 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" event={"ID":"ade6cd92-b99f-4799-b73d-878bece393d9","Type":"ContainerStarted","Data":"ca67ed23f49886b9bcf665be259dfb67db5e64dba70e6bcb81f1ed6474871082"} Nov 03 00:14:44.386796 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:44.386747 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" event={"ID":"135c8d1b-e9c2-4f5a-8b05-ea277767f83b","Type":"ContainerStarted","Data":"591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf"} Nov 03 00:14:44.386934 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:44.386824 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" event={"ID":"135c8d1b-e9c2-4f5a-8b05-ea277767f83b","Type":"ContainerStarted","Data":"21fbed81b8a35c773ee079b7cb566cf754ecd5cfbff676da915112578b32985f"} Nov 03 00:14:44.387988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:44.387965 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" event={"ID":"289daeb7-1acb-4aad-a229-af361e633761","Type":"ContainerStarted","Data":"8e46e7992b78f6e5f16a7b9552ef2358def0c30f7a1d0ac1a1f4510a806a74fc"} Nov 03 00:14:44.504254 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:44.504200 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" podStartSLOduration=2.5041858980000002 podStartE2EDuration="2.504185898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:44.502123244 +0000 UTC m=+567.785680931" watchObservedRunningTime="2026-11-03 00:14:44.504185898 +0000 UTC m=+567.787743618" Nov 03 00:14:45.145120 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.145081 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:45.152268 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.152241 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:45.370870 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.370823 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9"] Nov 03 00:14:45.374618 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.374589 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:45.396098 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.396022 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:45.398364 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.398340 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:14:45.426635 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.426598 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-01450ec4-serving-cert\"" Nov 03 00:14:45.432289 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.432249 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/800631aa-8772-4844-865a-b8f6c5bbd75e-openshift-service-ca\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:45.432458 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.432366 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-metrics-certs\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:45.432458 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.432423 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njqn\" (UniqueName: \"kubernetes.io/projected/800631aa-8772-4844-865a-b8f6c5bbd75e-kube-api-access-9njqn\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:45.432617 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.432467 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-maas-api-tls\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:45.433981 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.433698 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9"] Nov 03 00:14:45.533373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.533331 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/800631aa-8772-4844-865a-b8f6c5bbd75e-openshift-service-ca\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:45.533569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.533440 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-metrics-certs\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:45.533569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.533479 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njqn\" (UniqueName: \"kubernetes.io/projected/800631aa-8772-4844-865a-b8f6c5bbd75e-kube-api-access-9njqn\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:45.533569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.533515 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-maas-api-tls\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:45.533759 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:14:45.533703 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-serving-cert: secret "maas-api-e2e-ait-01450ec4-serving-cert" not found Nov 03 00:14:45.533852 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:14:45.533833 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-maas-api-tls podName:800631aa-8772-4844-865a-b8f6c5bbd75e nodeName:}" failed. No retries permitted until 2026-11-03 00:14:46.033809223 +0000 UTC m=+569.317366908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-maas-api-tls") pod "maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" (UID: "800631aa-8772-4844-865a-b8f6c5bbd75e") : secret "maas-api-e2e-ait-01450ec4-serving-cert" not found Nov 03 00:14:45.534212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.534188 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/800631aa-8772-4844-865a-b8f6c5bbd75e-openshift-service-ca\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:45.537707 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.537678 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-metrics-certs\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:45.593373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:45.593317 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njqn\" (UniqueName: \"kubernetes.io/projected/800631aa-8772-4844-865a-b8f6c5bbd75e-kube-api-access-9njqn\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:46.039257 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:46.039215 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-maas-api-tls\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:46.042318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:46.042288 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-maas-api-tls\") pod \"maas-api-e2e-ait-01450ec4-6467967b79-rmzp9\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:46.292274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:46.292195 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:46.685066 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:46.685039 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9"] Nov 03 00:14:46.687847 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:14:46.687822 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800631aa_8772_4844_865a_b8f6c5bbd75e.slice/crio-11253c55388ba95f6c28977ab646ba416a69f6d91c4561a8ada314be21df7522 WatchSource:0}: Error finding container 11253c55388ba95f6c28977ab646ba416a69f6d91c4561a8ada314be21df7522: Status 404 returned error can't find the container with id 11253c55388ba95f6c28977ab646ba416a69f6d91c4561a8ada314be21df7522 Nov 03 00:14:47.406838 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:47.406802 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" event={"ID":"289daeb7-1acb-4aad-a229-af361e633761","Type":"ContainerStarted","Data":"44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011"} Nov 03 00:14:47.407285 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:47.406902 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:14:47.408185 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:47.408164 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" event={"ID":"800631aa-8772-4844-865a-b8f6c5bbd75e","Type":"ContainerStarted","Data":"11253c55388ba95f6c28977ab646ba416a69f6d91c4561a8ada314be21df7522"} Nov 03 00:14:47.409655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:47.409636 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" event={"ID":"ade6cd92-b99f-4799-b73d-878bece393d9","Type":"ContainerStarted","Data":"5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6"} Nov 03 00:14:47.409900 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:47.409885 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:14:47.521946 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:47.521892 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" podStartSLOduration=1.418982352 podStartE2EDuration="4.521877566s" totalImagesPullingTime="3.102895214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:47.5195456 +0000 UTC m=+570.803103297" watchObservedRunningTime="2026-11-03 00:14:47.521877566 +0000 UTC m=+570.805435240" Nov 03 00:14:47.634877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:47.634825 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" podStartSLOduration=1.709815817 podStartE2EDuration="4.634805575s" totalImagesPullingTime="2.924989758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:47.632748714 +0000 UTC m=+570.916306387" watchObservedRunningTime="2026-11-03 00:14:47.634805575 +0000 UTC m=+570.918363250" Nov 03 00:14:47.956471 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:47.956402 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4"] Nov 03 00:14:47.959856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:47.959838 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:14:48.014383 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.014344 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-01450ec4\"" Nov 03 00:14:48.014547 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.014467 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-01450ec4-dockercfg-5jwd6\"" Nov 03 00:14:48.016899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.016833 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4"] Nov 03 00:14:48.057148 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.057108 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c6af80-9613-473c-95c8-eb8c31922178-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4\" (UID: \"b5c6af80-9613-473c-95c8-eb8c31922178\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:14:48.057314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.057220 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4pkn\" (UniqueName: \"kubernetes.io/projected/b5c6af80-9613-473c-95c8-eb8c31922178-kube-api-access-j4pkn\") pod \"payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4\" (UID: \"b5c6af80-9613-473c-95c8-eb8c31922178\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:14:48.110148 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.110113 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp"] Nov 03 00:14:48.113341 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.113325 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:14:48.158525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.158497 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrqxd\" (UniqueName: \"kubernetes.io/projected/41c4c48b-14af-44fa-8a6d-990972f28552-kube-api-access-xrqxd\") pod \"payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp\" (UID: \"41c4c48b-14af-44fa-8a6d-990972f28552\") " pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:14:48.158666 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.158553 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c6af80-9613-473c-95c8-eb8c31922178-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4\" (UID: \"b5c6af80-9613-473c-95c8-eb8c31922178\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:14:48.158666 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.158656 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c4c48b-14af-44fa-8a6d-990972f28552-plugins-config-volume\") pod \"payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp\" (UID: \"41c4c48b-14af-44fa-8a6d-990972f28552\") " pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:14:48.158806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.158688 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4pkn\" (UniqueName: \"kubernetes.io/projected/b5c6af80-9613-473c-95c8-eb8c31922178-kube-api-access-j4pkn\") pod \"payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4\" (UID: \"b5c6af80-9613-473c-95c8-eb8c31922178\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:14:48.159283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.159262 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c6af80-9613-473c-95c8-eb8c31922178-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4\" (UID: \"b5c6af80-9613-473c-95c8-eb8c31922178\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:14:48.170554 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.170530 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp"] Nov 03 00:14:48.215304 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.215238 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4pkn\" (UniqueName: \"kubernetes.io/projected/b5c6af80-9613-473c-95c8-eb8c31922178-kube-api-access-j4pkn\") pod \"payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4\" (UID: \"b5c6af80-9613-473c-95c8-eb8c31922178\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:14:48.259381 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.259354 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrqxd\" (UniqueName: \"kubernetes.io/projected/41c4c48b-14af-44fa-8a6d-990972f28552-kube-api-access-xrqxd\") pod \"payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp\" (UID: \"41c4c48b-14af-44fa-8a6d-990972f28552\") " pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:14:48.259510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.259432 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c4c48b-14af-44fa-8a6d-990972f28552-plugins-config-volume\") pod \"payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp\" (UID: \"41c4c48b-14af-44fa-8a6d-990972f28552\") " pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:14:48.260042 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.260026 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c4c48b-14af-44fa-8a6d-990972f28552-plugins-config-volume\") pod \"payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp\" (UID: \"41c4c48b-14af-44fa-8a6d-990972f28552\") " pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:14:48.273395 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.273375 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:14:48.320094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.320058 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrqxd\" (UniqueName: \"kubernetes.io/projected/41c4c48b-14af-44fa-8a6d-990972f28552-kube-api-access-xrqxd\") pod \"payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp\" (UID: \"41c4c48b-14af-44fa-8a6d-990972f28552\") " pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:14:48.415607 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.415567 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" event={"ID":"800631aa-8772-4844-865a-b8f6c5bbd75e","Type":"ContainerStarted","Data":"31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25"} Nov 03 00:14:48.416038 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.415863 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:48.426138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.426119 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:14:48.530155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.530099 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" podStartSLOduration=2.5915184 podStartE2EDuration="3.530083544s" totalImagesPullingTime="938.565144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:48.527877411 +0000 UTC m=+571.811435084" watchObservedRunningTime="2026-11-03 00:14:48.530083544 +0000 UTC m=+571.813641231" Nov 03 00:14:48.646626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.646595 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4"] Nov 03 00:14:48.648054 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:14:48.648030 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c6af80_9613_473c_95c8_eb8c31922178.slice/crio-59a59e472b401ab647304f4f3703ec610d4d01a8bbe5de9ca6d9f3a1982420cd WatchSource:0}: Error finding container 59a59e472b401ab647304f4f3703ec610d4d01a8bbe5de9ca6d9f3a1982420cd: Status 404 returned error can't find the container with id 59a59e472b401ab647304f4f3703ec610d4d01a8bbe5de9ca6d9f3a1982420cd Nov 03 00:14:48.800138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:48.800111 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp"] Nov 03 00:14:48.802241 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:14:48.802215 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c4c48b_14af_44fa_8a6d_990972f28552.slice/crio-f1897c70395f78a8619b16a0d6dfe235b3b61ba7e1c171b3b51f005f8a7c08d1 WatchSource:0}: Error finding container f1897c70395f78a8619b16a0d6dfe235b3b61ba7e1c171b3b51f005f8a7c08d1: Status 404 returned error can't find the container with id f1897c70395f78a8619b16a0d6dfe235b3b61ba7e1c171b3b51f005f8a7c08d1 Nov 03 00:14:49.421799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:49.421738 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" event={"ID":"41c4c48b-14af-44fa-8a6d-990972f28552","Type":"ContainerStarted","Data":"b57cacf60d8ba04fe8d271dc9bb31d1bfab3ad0b154beac8c822fe851b328116"} Nov 03 00:14:49.422248 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:49.421824 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" event={"ID":"41c4c48b-14af-44fa-8a6d-990972f28552","Type":"ContainerStarted","Data":"f1897c70395f78a8619b16a0d6dfe235b3b61ba7e1c171b3b51f005f8a7c08d1"} Nov 03 00:14:49.422248 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:49.421851 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:14:49.423325 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:49.423295 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" event={"ID":"b5c6af80-9613-473c-95c8-eb8c31922178","Type":"ContainerStarted","Data":"10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd"} Nov 03 00:14:49.423450 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:49.423336 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" event={"ID":"b5c6af80-9613-473c-95c8-eb8c31922178","Type":"ContainerStarted","Data":"59a59e472b401ab647304f4f3703ec610d4d01a8bbe5de9ca6d9f3a1982420cd"} Nov 03 00:14:49.423450 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:49.423377 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:14:49.534727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:49.534674 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" podStartSLOduration=1.534661642 podStartE2EDuration="1.534661642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:49.533655779 +0000 UTC m=+572.817213453" watchObservedRunningTime="2026-11-03 00:14:49.534661642 +0000 UTC m=+572.818219315" Nov 03 00:14:49.647884 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:49.647835 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" podStartSLOduration=2.647819031 podStartE2EDuration="2.647819031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:49.64607637 +0000 UTC m=+572.929634044" watchObservedRunningTime="2026-11-03 00:14:49.647819031 +0000 UTC m=+572.931376760" Nov 03 00:14:50.453080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.453046 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn"] Nov 03 00:14:50.456738 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.456716 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:14:50.480418 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.480379 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8sch\" (UniqueName: \"kubernetes.io/projected/bba22536-45fa-4f07-b43f-1a774948b0d7-kube-api-access-r8sch\") pod \"payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn\" (UID: \"bba22536-45fa-4f07-b43f-1a774948b0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:14:50.480658 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.480472 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba22536-45fa-4f07-b43f-1a774948b0d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn\" (UID: \"bba22536-45fa-4f07-b43f-1a774948b0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:14:50.513974 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.513949 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-12537192-dockercfg-hl6pv\"" Nov 03 00:14:50.514322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.514297 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-12537192\"" Nov 03 00:14:50.565974 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.565937 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn"] Nov 03 00:14:50.576740 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.576710 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw"] Nov 03 00:14:50.580056 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.580038 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.581117 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.581091 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8sch\" (UniqueName: \"kubernetes.io/projected/bba22536-45fa-4f07-b43f-1a774948b0d7-kube-api-access-r8sch\") pod \"payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn\" (UID: \"bba22536-45fa-4f07-b43f-1a774948b0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:14:50.581203 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.581137 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba22536-45fa-4f07-b43f-1a774948b0d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn\" (UID: \"bba22536-45fa-4f07-b43f-1a774948b0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:14:50.582059 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.582033 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba22536-45fa-4f07-b43f-1a774948b0d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn\" (UID: \"bba22536-45fa-4f07-b43f-1a774948b0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:14:50.599884 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.599862 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt"] Nov 03 00:14:50.603183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.603168 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:14:50.631693 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.631671 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-12537192-serving-cert\"" Nov 03 00:14:50.638594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.638567 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw"] Nov 03 00:14:50.642010 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.641985 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8sch\" (UniqueName: \"kubernetes.io/projected/bba22536-45fa-4f07-b43f-1a774948b0d7-kube-api-access-r8sch\") pod \"payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn\" (UID: \"bba22536-45fa-4f07-b43f-1a774948b0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:14:50.662244 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.662024 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt"] Nov 03 00:14:50.682590 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.682553 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcf6502-70ee-4afc-a6ca-040130d60622-openshift-service-ca\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.682803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.682603 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-12537192-6f58444f67-q8spt\" (UID: \"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2\") " pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:14:50.682803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.682725 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d2jd\" (UniqueName: \"kubernetes.io/projected/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-kube-api-access-2d2jd\") pod \"payload-processing-e2e-ait-12537192-6f58444f67-q8spt\" (UID: \"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2\") " pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:14:50.682803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.682791 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-maas-api-tls\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.682926 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.682913 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76mzc\" (UniqueName: \"kubernetes.io/projected/2fcf6502-70ee-4afc-a6ca-040130d60622-kube-api-access-76mzc\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.682963 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.682936 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-metrics-certs\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.770556 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.770487 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:14:50.783742 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.783719 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d2jd\" (UniqueName: \"kubernetes.io/projected/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-kube-api-access-2d2jd\") pod \"payload-processing-e2e-ait-12537192-6f58444f67-q8spt\" (UID: \"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2\") " pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:14:50.783832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.783786 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-maas-api-tls\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.783882 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.783840 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76mzc\" (UniqueName: \"kubernetes.io/projected/2fcf6502-70ee-4afc-a6ca-040130d60622-kube-api-access-76mzc\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.783882 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.783856 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-metrics-certs\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.783980 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.783896 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcf6502-70ee-4afc-a6ca-040130d60622-openshift-service-ca\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.783980 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.783951 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-12537192-6f58444f67-q8spt\" (UID: \"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2\") " pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:14:50.783980 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:14:50.783962 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-12537192-serving-cert: secret "maas-api-e2e-ait-12537192-serving-cert" not found Nov 03 00:14:50.784125 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:14:50.784031 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-maas-api-tls podName:2fcf6502-70ee-4afc-a6ca-040130d60622 nodeName:}" failed. No retries permitted until 2026-11-03 00:14:51.284009629 +0000 UTC m=+574.567567297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-maas-api-tls") pod "maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" (UID: "2fcf6502-70ee-4afc-a6ca-040130d60622") : secret "maas-api-e2e-ait-12537192-serving-cert" not found Nov 03 00:14:50.784682 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.784603 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcf6502-70ee-4afc-a6ca-040130d60622-openshift-service-ca\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.784682 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.784656 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-12537192-6f58444f67-q8spt\" (UID: \"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2\") " pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:14:50.786514 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.786480 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-metrics-certs\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.844601 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.843853 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76mzc\" (UniqueName: \"kubernetes.io/projected/2fcf6502-70ee-4afc-a6ca-040130d60622-kube-api-access-76mzc\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:50.851869 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.851840 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d2jd\" (UniqueName: \"kubernetes.io/projected/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-kube-api-access-2d2jd\") pod \"payload-processing-e2e-ait-12537192-6f58444f67-q8spt\" (UID: \"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2\") " pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:14:50.916668 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:50.916642 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:14:51.146332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.146308 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn"] Nov 03 00:14:51.147609 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:14:51.147582 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba22536_45fa_4f07_b43f_1a774948b0d7.slice/crio-2894a3358ef895213ef946b895ea0c5ae8ad87520ea4633f7e1b4d215cfe7dcd WatchSource:0}: Error finding container 2894a3358ef895213ef946b895ea0c5ae8ad87520ea4633f7e1b4d215cfe7dcd: Status 404 returned error can't find the container with id 2894a3358ef895213ef946b895ea0c5ae8ad87520ea4633f7e1b4d215cfe7dcd Nov 03 00:14:51.286007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.285983 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt"] Nov 03 00:14:51.288099 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:14:51.288071 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3757594c_a8fc_4fc1_b23d_a4d01b62f8e2.slice/crio-57c6a0ed615971c9772717343bf61b5a3f7a8e41ffd331f801fc41daa0730b08 WatchSource:0}: Error finding container 57c6a0ed615971c9772717343bf61b5a3f7a8e41ffd331f801fc41daa0730b08: Status 404 returned error can't find the container with id 57c6a0ed615971c9772717343bf61b5a3f7a8e41ffd331f801fc41daa0730b08 Nov 03 00:14:51.289467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.289437 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-maas-api-tls\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:51.292276 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.292254 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-maas-api-tls\") pod \"maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:51.435282 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.435187 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" event={"ID":"bba22536-45fa-4f07-b43f-1a774948b0d7","Type":"ContainerStarted","Data":"6cab07dbf199c49deee55479aa143c880a136897b5b848ba61043e47b40cf606"} Nov 03 00:14:51.435282 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.435236 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" event={"ID":"bba22536-45fa-4f07-b43f-1a774948b0d7","Type":"ContainerStarted","Data":"2894a3358ef895213ef946b895ea0c5ae8ad87520ea4633f7e1b4d215cfe7dcd"} Nov 03 00:14:51.435513 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.435397 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:14:51.436981 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.436954 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" event={"ID":"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2","Type":"ContainerStarted","Data":"3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462"} Nov 03 00:14:51.437128 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.437002 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" event={"ID":"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2","Type":"ContainerStarted","Data":"57c6a0ed615971c9772717343bf61b5a3f7a8e41ffd331f801fc41daa0730b08"} Nov 03 00:14:51.437128 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.437031 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:14:51.495976 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.495945 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:51.547534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.547415 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" podStartSLOduration=1.547396239 podStartE2EDuration="1.547396239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:51.54688183 +0000 UTC m=+574.830439504" watchObservedRunningTime="2026-11-03 00:14:51.547396239 +0000 UTC m=+574.830953927" Nov 03 00:14:51.678173 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.678125 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" podStartSLOduration=1.6781109889999999 podStartE2EDuration="1.678110989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:51.675711258 +0000 UTC m=+574.959268943" watchObservedRunningTime="2026-11-03 00:14:51.678110989 +0000 UTC m=+574.961668663" Nov 03 00:14:51.867542 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:51.867423 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw"] Nov 03 00:14:51.869884 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:14:51.869861 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcf6502_70ee_4afc_a6ca_040130d60622.slice/crio-497f404a3b0795ce5b0a0f18940eb98248d1eca9b228ced4dbff31e14647ca26 WatchSource:0}: Error finding container 497f404a3b0795ce5b0a0f18940eb98248d1eca9b228ced4dbff31e14647ca26: Status 404 returned error can't find the container with id 497f404a3b0795ce5b0a0f18940eb98248d1eca9b228ced4dbff31e14647ca26 Nov 03 00:14:52.443494 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:52.443457 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" event={"ID":"2fcf6502-70ee-4afc-a6ca-040130d60622","Type":"ContainerStarted","Data":"497f404a3b0795ce5b0a0f18940eb98248d1eca9b228ced4dbff31e14647ca26"} Nov 03 00:14:53.075906 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.075833 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6"] Nov 03 00:14:53.079356 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.079339 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:14:53.108579 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.108545 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e31dcbe-63d2-49b9-84da-ee9d0914d377-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6\" (UID: \"8e31dcbe-63d2-49b9-84da-ee9d0914d377\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:14:53.108579 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.108581 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q2dp\" (UniqueName: \"kubernetes.io/projected/8e31dcbe-63d2-49b9-84da-ee9d0914d377-kube-api-access-9q2dp\") pod \"payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6\" (UID: \"8e31dcbe-63d2-49b9-84da-ee9d0914d377\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:14:53.130882 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.130853 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c6f27688-dockercfg-kcx7j\"" Nov 03 00:14:53.180446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.180415 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c6f27688\"" Nov 03 00:14:53.183012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.182986 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6"] Nov 03 00:14:53.210023 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.209995 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e31dcbe-63d2-49b9-84da-ee9d0914d377-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6\" (UID: \"8e31dcbe-63d2-49b9-84da-ee9d0914d377\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:14:53.210155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.210032 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q2dp\" (UniqueName: \"kubernetes.io/projected/8e31dcbe-63d2-49b9-84da-ee9d0914d377-kube-api-access-9q2dp\") pod \"payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6\" (UID: \"8e31dcbe-63d2-49b9-84da-ee9d0914d377\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:14:53.210717 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.210698 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e31dcbe-63d2-49b9-84da-ee9d0914d377-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6\" (UID: \"8e31dcbe-63d2-49b9-84da-ee9d0914d377\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:14:53.236372 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.236344 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp"] Nov 03 00:14:53.240377 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.240358 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:14:53.266049 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.266021 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q2dp\" (UniqueName: \"kubernetes.io/projected/8e31dcbe-63d2-49b9-84da-ee9d0914d377-kube-api-access-9q2dp\") pod \"payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6\" (UID: \"8e31dcbe-63d2-49b9-84da-ee9d0914d377\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:14:53.297253 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.297224 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp"] Nov 03 00:14:53.310911 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.310884 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccfrd\" (UniqueName: \"kubernetes.io/projected/bc16e901-e277-46c6-96f7-2cf983dd9e03-kube-api-access-ccfrd\") pod \"payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp\" (UID: \"bc16e901-e277-46c6-96f7-2cf983dd9e03\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:14:53.311058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.311028 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc16e901-e277-46c6-96f7-2cf983dd9e03-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp\" (UID: \"bc16e901-e277-46c6-96f7-2cf983dd9e03\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:14:53.392132 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.392052 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:14:53.412385 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.412356 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc16e901-e277-46c6-96f7-2cf983dd9e03-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp\" (UID: \"bc16e901-e277-46c6-96f7-2cf983dd9e03\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:14:53.412504 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.412435 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccfrd\" (UniqueName: \"kubernetes.io/projected/bc16e901-e277-46c6-96f7-2cf983dd9e03-kube-api-access-ccfrd\") pod \"payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp\" (UID: \"bc16e901-e277-46c6-96f7-2cf983dd9e03\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:14:53.413182 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.413158 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc16e901-e277-46c6-96f7-2cf983dd9e03-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp\" (UID: \"bc16e901-e277-46c6-96f7-2cf983dd9e03\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:14:53.453264 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.453228 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" event={"ID":"2fcf6502-70ee-4afc-a6ca-040130d60622","Type":"ContainerStarted","Data":"2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80"} Nov 03 00:14:53.454030 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.454004 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:14:53.468582 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.468561 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccfrd\" (UniqueName: \"kubernetes.io/projected/bc16e901-e277-46c6-96f7-2cf983dd9e03-kube-api-access-ccfrd\") pod \"payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp\" (UID: \"bc16e901-e277-46c6-96f7-2cf983dd9e03\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:14:53.553577 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.553545 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:14:53.569558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.569508 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" podStartSLOduration=2.78043524 podStartE2EDuration="3.569493046s" totalImagesPullingTime="789.057806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:53.566978699 +0000 UTC m=+576.850536391" watchObservedRunningTime="2026-11-03 00:14:53.569493046 +0000 UTC m=+576.853051125" Nov 03 00:14:53.760566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.760541 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6"] Nov 03 00:14:53.761736 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:14:53.761706 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e31dcbe_63d2_49b9_84da_ee9d0914d377.slice/crio-604b5e7deca62362a870ebb1407015686bd2c3696a5d80e69c5a46e1a5b53103 WatchSource:0}: Error finding container 604b5e7deca62362a870ebb1407015686bd2c3696a5d80e69c5a46e1a5b53103: Status 404 returned error can't find the container with id 604b5e7deca62362a870ebb1407015686bd2c3696a5d80e69c5a46e1a5b53103 Nov 03 00:14:53.924406 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:53.924378 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp"] Nov 03 00:14:53.926009 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:14:53.925981 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc16e901_e277_46c6_96f7_2cf983dd9e03.slice/crio-86d6fa4ca9d57aa61bf78bcd882606d11980b3ee1c7ba968603ebff252bca06b WatchSource:0}: Error finding container 86d6fa4ca9d57aa61bf78bcd882606d11980b3ee1c7ba968603ebff252bca06b: Status 404 returned error can't find the container with id 86d6fa4ca9d57aa61bf78bcd882606d11980b3ee1c7ba968603ebff252bca06b Nov 03 00:14:54.429476 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:54.429447 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:14:54.461218 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:54.461183 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" event={"ID":"8e31dcbe-63d2-49b9-84da-ee9d0914d377","Type":"ContainerStarted","Data":"271453b76f0e795f8bfd8d00b03bdecc127b7f89283e1bc9ad279904af29aa8f"} Nov 03 00:14:54.461406 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:54.461246 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:14:54.461406 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:54.461267 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" event={"ID":"8e31dcbe-63d2-49b9-84da-ee9d0914d377","Type":"ContainerStarted","Data":"604b5e7deca62362a870ebb1407015686bd2c3696a5d80e69c5a46e1a5b53103"} Nov 03 00:14:54.462659 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:54.462636 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" event={"ID":"bc16e901-e277-46c6-96f7-2cf983dd9e03","Type":"ContainerStarted","Data":"32ac2f4381308c55934091d26fd0657d9afd703547e025190d87c9b3c5247688"} Nov 03 00:14:54.462800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:54.462672 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" event={"ID":"bc16e901-e277-46c6-96f7-2cf983dd9e03","Type":"ContainerStarted","Data":"86d6fa4ca9d57aa61bf78bcd882606d11980b3ee1c7ba968603ebff252bca06b"} Nov 03 00:14:54.462800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:54.462706 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:14:54.657052 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:54.656986 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" podStartSLOduration=1.656971424 podStartE2EDuration="1.656971424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:54.654973047 +0000 UTC m=+577.938530722" watchObservedRunningTime="2026-11-03 00:14:54.656971424 +0000 UTC m=+577.940529097" Nov 03 00:14:54.766864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:54.766739 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" podStartSLOduration=1.766723195 podStartE2EDuration="1.766723195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:54.766150672 +0000 UTC m=+578.049708346" watchObservedRunningTime="2026-11-03 00:14:54.766723195 +0000 UTC m=+578.050280871" Nov 03 00:14:58.418689 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:58.418660 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:14:58.420962 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:14:58.418731 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:14:58.656185 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:14:58.656149 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:15:00.431803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:00.431715 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:15:00.432310 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:00.431908 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:15:00.474136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:00.474112 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:15:02.446761 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:02.446738 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:15:02.447197 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:02.446817 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:15:02.896324 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:02.896293 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c"] Nov 03 00:15:02.900012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:02.899991 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:02.952435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:02.952414 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-168f474d-dockercfg-7wngg\"" Nov 03 00:15:02.953130 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:02.953104 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-168f474d\"" Nov 03 00:15:02.961934 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:02.961904 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c"] Nov 03 00:15:03.006889 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:03.006858 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4sg\" (UniqueName: \"kubernetes.io/projected/5a591f05-0c35-4309-a2e2-f43e898c6918-kube-api-access-mv4sg\") pod \"payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c\" (UID: \"5a591f05-0c35-4309-a2e2-f43e898c6918\") " pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:03.007030 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:03.006898 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a591f05-0c35-4309-a2e2-f43e898c6918-plugins-config-volume\") pod \"payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c\" (UID: \"5a591f05-0c35-4309-a2e2-f43e898c6918\") " pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:03.108395 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:03.108362 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4sg\" (UniqueName: \"kubernetes.io/projected/5a591f05-0c35-4309-a2e2-f43e898c6918-kube-api-access-mv4sg\") pod \"payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c\" (UID: \"5a591f05-0c35-4309-a2e2-f43e898c6918\") " pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:03.108559 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:03.108404 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a591f05-0c35-4309-a2e2-f43e898c6918-plugins-config-volume\") pod \"payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c\" (UID: \"5a591f05-0c35-4309-a2e2-f43e898c6918\") " pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:03.109112 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:03.109093 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a591f05-0c35-4309-a2e2-f43e898c6918-plugins-config-volume\") pod \"payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c\" (UID: \"5a591f05-0c35-4309-a2e2-f43e898c6918\") " pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:03.167555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:03.167479 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4sg\" (UniqueName: \"kubernetes.io/projected/5a591f05-0c35-4309-a2e2-f43e898c6918-kube-api-access-mv4sg\") pod \"payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c\" (UID: \"5a591f05-0c35-4309-a2e2-f43e898c6918\") " pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:03.212552 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:03.212524 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:03.589009 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:03.588988 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c"] Nov 03 00:15:03.591097 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:15:03.591067 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a591f05_0c35_4309_a2e2_f43e898c6918.slice/crio-fd9aa3b4e68d97b2867027b77a8986b4953b444a9c71b49a3041532e00141ce3 WatchSource:0}: Error finding container fd9aa3b4e68d97b2867027b77a8986b4953b444a9c71b49a3041532e00141ce3: Status 404 returned error can't find the container with id fd9aa3b4e68d97b2867027b77a8986b4953b444a9c71b49a3041532e00141ce3 Nov 03 00:15:04.514988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:04.514925 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" event={"ID":"5a591f05-0c35-4309-a2e2-f43e898c6918","Type":"ContainerStarted","Data":"3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b"} Nov 03 00:15:04.514988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:04.514971 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" event={"ID":"5a591f05-0c35-4309-a2e2-f43e898c6918","Type":"ContainerStarted","Data":"fd9aa3b4e68d97b2867027b77a8986b4953b444a9c71b49a3041532e00141ce3"} Nov 03 00:15:04.515274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:04.514998 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:04.631048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:04.631000 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" podStartSLOduration=2.630985517 podStartE2EDuration="2.630985517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:04.629108601 +0000 UTC m=+587.912666278" watchObservedRunningTime="2026-11-03 00:15:04.630985517 +0000 UTC m=+587.914543191" Nov 03 00:15:05.470988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.470960 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:15:05.471167 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.471028 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:15:05.702742 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.702709 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk"] Nov 03 00:15:05.706301 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.706286 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:05.762284 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.762217 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-168f474d-serving-cert\"" Nov 03 00:15:05.778124 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.778096 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk"] Nov 03 00:15:05.833195 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.833162 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-maas-api-tls\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:05.833374 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.833212 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-metrics-certs\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:05.833442 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.833388 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80fd667f-313f-429c-95e1-a8856cca4122-openshift-service-ca\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:05.833500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.833437 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65d56\" (UniqueName: \"kubernetes.io/projected/80fd667f-313f-429c-95e1-a8856cca4122-kube-api-access-65d56\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:05.934448 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.934418 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-metrics-certs\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:05.934648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.934507 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80fd667f-313f-429c-95e1-a8856cca4122-openshift-service-ca\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:05.934648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.934535 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65d56\" (UniqueName: \"kubernetes.io/projected/80fd667f-313f-429c-95e1-a8856cca4122-kube-api-access-65d56\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:05.934648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.934592 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-maas-api-tls\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:05.934872 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:15:05.934714 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-serving-cert: secret "maas-api-e2e-derive-168f474d-serving-cert" not found Nov 03 00:15:05.934872 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:15:05.934800 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-maas-api-tls podName:80fd667f-313f-429c-95e1-a8856cca4122 nodeName:}" failed. No retries permitted until 2026-11-03 00:15:06.434764453 +0000 UTC m=+589.718322110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-maas-api-tls") pod "maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" (UID: "80fd667f-313f-429c-95e1-a8856cca4122") : secret "maas-api-e2e-derive-168f474d-serving-cert" not found Nov 03 00:15:05.935370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.935307 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80fd667f-313f-429c-95e1-a8856cca4122-openshift-service-ca\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:05.937190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.937167 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-metrics-certs\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:05.992667 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:05.992635 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65d56\" (UniqueName: \"kubernetes.io/projected/80fd667f-313f-429c-95e1-a8856cca4122-kube-api-access-65d56\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:06.440059 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:06.440022 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-maas-api-tls\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:06.442516 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:06.442498 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-maas-api-tls\") pod \"maas-api-e2e-derive-168f474d-64c86d64d6-bscrk\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:06.618651 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:06.618611 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:06.988471 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:06.988443 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk"] Nov 03 00:15:06.990568 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:15:06.990540 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fd667f_313f_429c_95e1_a8856cca4122.slice/crio-70157279089f2944a1a6392f3ee47356c7d48f9d35a5c75036ef82b1f5065656 WatchSource:0}: Error finding container 70157279089f2944a1a6392f3ee47356c7d48f9d35a5c75036ef82b1f5065656: Status 404 returned error can't find the container with id 70157279089f2944a1a6392f3ee47356c7d48f9d35a5c75036ef82b1f5065656 Nov 03 00:15:07.532294 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:07.532260 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" event={"ID":"80fd667f-313f-429c-95e1-a8856cca4122","Type":"ContainerStarted","Data":"70157279089f2944a1a6392f3ee47356c7d48f9d35a5c75036ef82b1f5065656"} Nov 03 00:15:08.538953 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:08.538913 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" event={"ID":"80fd667f-313f-429c-95e1-a8856cca4122","Type":"ContainerStarted","Data":"e885a597619b60fc97ec96376ec957da7789ba7d8adbfb6356ad247f751273ef"} Nov 03 00:15:08.539359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:08.539073 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:08.655487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:08.655425 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" podStartSLOduration=2.6982474 podStartE2EDuration="3.655406515s" totalImagesPullingTime="957.159115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:08.653863336 +0000 UTC m=+591.937421011" watchObservedRunningTime="2026-11-03 00:15:08.655406515 +0000 UTC m=+591.938964191" Nov 03 00:15:14.549585 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:14.549557 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:15.523278 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:15.523248 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:25.628021 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:25.627987 4304 generic.go:356] "Generic (PLEG): container finished" podID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerID="591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf" exitCode=0 Nov 03 00:15:25.628476 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:25.628047 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" event={"ID":"135c8d1b-e9c2-4f5a-8b05-ea277767f83b","Type":"ContainerDied","Data":"591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf"} Nov 03 00:15:25.628476 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:25.628389 4304 scope.go:122] "RemoveContainer" containerID="591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf" Nov 03 00:15:25.629534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:25.629510 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:25.629634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:25.629575 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:25.629634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:25.629608 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:26.652221 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:26.652185 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" event={"ID":"135c8d1b-e9c2-4f5a-8b05-ea277767f83b","Type":"ContainerStarted","Data":"b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba"} Nov 03 00:15:27.146153 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:27.145997 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:15:27.147009 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:27.146981 4304 patch_prober.go:29] interesting pod/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:15:27.147147 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:27.147079 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 00:15:28.147732 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:28.147696 4304 patch_prober.go:29] interesting pod/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:15:28.148242 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:28.147803 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 00:15:28.670445 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:15:28.670411 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:15:29.146166 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:29.146129 4304 patch_prober.go:29] interesting pod/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:15:29.146365 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:29.146184 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 00:15:30.146164 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:30.146126 4304 patch_prober.go:29] interesting pod/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:15:30.146577 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:30.146178 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 00:15:30.396330 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:30.396227 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:15:31.146255 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:31.146207 4304 patch_prober.go:29] interesting pod/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:15:31.146668 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:31.146279 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 03 00:15:32.148905 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:32.148877 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:15:32.149842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:32.149826 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:15:38.608581 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:38.608544 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp"] Nov 03 00:15:38.714958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:38.714924 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4"] Nov 03 00:15:42.491590 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:42.491556 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt"] Nov 03 00:15:42.601406 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:42.601367 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn"] Nov 03 00:15:43.609737 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:43.609678 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" podUID="41c4c48b-14af-44fa-8a6d-990972f28552" containerName="payload-processing" containerID="cri-o://b57cacf60d8ba04fe8d271dc9bb31d1bfab3ad0b154beac8c822fe851b328116" gracePeriod=25 Nov 03 00:15:43.716251 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:43.716191 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" podUID="b5c6af80-9613-473c-95c8-eb8c31922178" containerName="payload-pre-processing" containerID="cri-o://10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd" gracePeriod=25 Nov 03 00:15:43.747845 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:43.747816 4304 generic.go:356] "Generic (PLEG): container finished" podID="41c4c48b-14af-44fa-8a6d-990972f28552" containerID="b57cacf60d8ba04fe8d271dc9bb31d1bfab3ad0b154beac8c822fe851b328116" exitCode=0 Nov 03 00:15:43.747991 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:43.747881 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" event={"ID":"41c4c48b-14af-44fa-8a6d-990972f28552","Type":"ContainerDied","Data":"b57cacf60d8ba04fe8d271dc9bb31d1bfab3ad0b154beac8c822fe851b328116"} Nov 03 00:15:43.870295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:43.870238 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:15:44.029430 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.029404 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c4c48b-14af-44fa-8a6d-990972f28552-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c4c48b-14af-44fa-8a6d-990972f28552-plugins-config-volume\") pod \"41c4c48b-14af-44fa-8a6d-990972f28552\" (UID: \"41c4c48b-14af-44fa-8a6d-990972f28552\") " Nov 03 00:15:44.029551 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.029543 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c4c48b-14af-44fa-8a6d-990972f28552-kube-api-access-xrqxd\" (UniqueName: \"kubernetes.io/projected/41c4c48b-14af-44fa-8a6d-990972f28552-kube-api-access-xrqxd\") pod \"41c4c48b-14af-44fa-8a6d-990972f28552\" (UID: \"41c4c48b-14af-44fa-8a6d-990972f28552\") " Nov 03 00:15:44.029695 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.029674 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c4c48b-14af-44fa-8a6d-990972f28552-plugins-config-volume" pod "41c4c48b-14af-44fa-8a6d-990972f28552" (UID: "41c4c48b-14af-44fa-8a6d-990972f28552"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:44.029840 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.029825 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c4c48b-14af-44fa-8a6d-990972f28552-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:44.030417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.030402 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:15:44.031723 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.031704 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c4c48b-14af-44fa-8a6d-990972f28552-kube-api-access-xrqxd" pod "41c4c48b-14af-44fa-8a6d-990972f28552" (UID: "41c4c48b-14af-44fa-8a6d-990972f28552"). InnerVolumeSpecName "kube-api-access-xrqxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:44.130559 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.130468 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c6af80-9613-473c-95c8-eb8c31922178-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c6af80-9613-473c-95c8-eb8c31922178-plugins-config-volume\") pod \"b5c6af80-9613-473c-95c8-eb8c31922178\" (UID: \"b5c6af80-9613-473c-95c8-eb8c31922178\") " Nov 03 00:15:44.130705 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.130619 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c6af80-9613-473c-95c8-eb8c31922178-kube-api-access-j4pkn\" (UniqueName: \"kubernetes.io/projected/b5c6af80-9613-473c-95c8-eb8c31922178-kube-api-access-j4pkn\") pod \"b5c6af80-9613-473c-95c8-eb8c31922178\" (UID: \"b5c6af80-9613-473c-95c8-eb8c31922178\") " Nov 03 00:15:44.130868 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.130855 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrqxd\" (UniqueName: \"kubernetes.io/projected/41c4c48b-14af-44fa-8a6d-990972f28552-kube-api-access-xrqxd\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:44.130943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.130918 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c6af80-9613-473c-95c8-eb8c31922178-plugins-config-volume" pod "b5c6af80-9613-473c-95c8-eb8c31922178" (UID: "b5c6af80-9613-473c-95c8-eb8c31922178"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:44.132804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.132763 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c6af80-9613-473c-95c8-eb8c31922178-kube-api-access-j4pkn" pod "b5c6af80-9613-473c-95c8-eb8c31922178" (UID: "b5c6af80-9613-473c-95c8-eb8c31922178"). InnerVolumeSpecName "kube-api-access-j4pkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:44.231943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.231907 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c6af80-9613-473c-95c8-eb8c31922178-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:44.231943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.231938 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4pkn\" (UniqueName: \"kubernetes.io/projected/b5c6af80-9613-473c-95c8-eb8c31922178-kube-api-access-j4pkn\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:44.464879 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.464850 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9"] Nov 03 00:15:44.465058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.465037 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" podUID="800631aa-8772-4844-865a-b8f6c5bbd75e" containerName="maas-api" containerID="cri-o://31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25" gracePeriod=30 Nov 03 00:15:44.719428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.719366 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:15:44.754100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.754073 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" Nov 03 00:15:44.754235 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.754104 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp" event={"ID":"41c4c48b-14af-44fa-8a6d-990972f28552","Type":"ContainerDied","Data":"f1897c70395f78a8619b16a0d6dfe235b3b61ba7e1c171b3b51f005f8a7c08d1"} Nov 03 00:15:44.754235 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.754172 4304 scope.go:122] "RemoveContainer" containerID="b57cacf60d8ba04fe8d271dc9bb31d1bfab3ad0b154beac8c822fe851b328116" Nov 03 00:15:44.755468 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.755445 4304 generic.go:356] "Generic (PLEG): container finished" podID="800631aa-8772-4844-865a-b8f6c5bbd75e" containerID="31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25" exitCode=0 Nov 03 00:15:44.755590 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.755476 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" event={"ID":"800631aa-8772-4844-865a-b8f6c5bbd75e","Type":"ContainerDied","Data":"31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25"} Nov 03 00:15:44.755590 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.755497 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" Nov 03 00:15:44.755590 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.755520 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9" event={"ID":"800631aa-8772-4844-865a-b8f6c5bbd75e","Type":"ContainerDied","Data":"11253c55388ba95f6c28977ab646ba416a69f6d91c4561a8ada314be21df7522"} Nov 03 00:15:44.756922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.756897 4304 generic.go:356] "Generic (PLEG): container finished" podID="b5c6af80-9613-473c-95c8-eb8c31922178" containerID="10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd" exitCode=0 Nov 03 00:15:44.757047 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.756938 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" Nov 03 00:15:44.757047 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.756957 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" event={"ID":"b5c6af80-9613-473c-95c8-eb8c31922178","Type":"ContainerDied","Data":"10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd"} Nov 03 00:15:44.757047 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.756997 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4" event={"ID":"b5c6af80-9613-473c-95c8-eb8c31922178","Type":"ContainerDied","Data":"59a59e472b401ab647304f4f3703ec610d4d01a8bbe5de9ca6d9f3a1982420cd"} Nov 03 00:15:44.764195 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.764177 4304 scope.go:122] "RemoveContainer" containerID="31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25" Nov 03 00:15:44.772254 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.772239 4304 scope.go:122] "RemoveContainer" containerID="31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25" Nov 03 00:15:44.772524 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:15:44.772505 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25\": container with ID starting with 31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25 not found: ID does not exist" containerID="31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25" Nov 03 00:15:44.772620 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.772530 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25"} err="failed to get container status \"31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25\": rpc error: code = NotFound desc = could not find container \"31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25\": container with ID starting with 31490131ad3c425cce0077341cbbaf9d2c20f877b0c66a82478cd66c31ef0e25 not found: ID does not exist" Nov 03 00:15:44.772620 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.772545 4304 scope.go:122] "RemoveContainer" containerID="10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd" Nov 03 00:15:44.780224 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.780211 4304 scope.go:122] "RemoveContainer" containerID="10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd" Nov 03 00:15:44.780459 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:15:44.780443 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd\": container with ID starting with 10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd not found: ID does not exist" containerID="10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd" Nov 03 00:15:44.780509 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.780465 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd"} err="failed to get container status \"10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd\": rpc error: code = NotFound desc = could not find container \"10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd\": container with ID starting with 10436eb05cf1ad04db29dcd19d42104cd23180fed07ec94be10d26b16d5799bd not found: ID does not exist" Nov 03 00:15:44.838257 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.838226 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-maas-api-tls\") pod \"800631aa-8772-4844-865a-b8f6c5bbd75e\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " Nov 03 00:15:44.838405 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.838324 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800631aa-8772-4844-865a-b8f6c5bbd75e-kube-api-access-9njqn\" (UniqueName: \"kubernetes.io/projected/800631aa-8772-4844-865a-b8f6c5bbd75e-kube-api-access-9njqn\") pod \"800631aa-8772-4844-865a-b8f6c5bbd75e\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " Nov 03 00:15:44.838405 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.838386 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-metrics-certs\") pod \"800631aa-8772-4844-865a-b8f6c5bbd75e\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " Nov 03 00:15:44.838530 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.838459 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800631aa-8772-4844-865a-b8f6c5bbd75e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/800631aa-8772-4844-865a-b8f6c5bbd75e-openshift-service-ca\") pod \"800631aa-8772-4844-865a-b8f6c5bbd75e\" (UID: \"800631aa-8772-4844-865a-b8f6c5bbd75e\") " Nov 03 00:15:44.839064 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.839041 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800631aa-8772-4844-865a-b8f6c5bbd75e-openshift-service-ca" pod "800631aa-8772-4844-865a-b8f6c5bbd75e" (UID: "800631aa-8772-4844-865a-b8f6c5bbd75e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:44.841494 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.841463 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-maas-api-tls" pod "800631aa-8772-4844-865a-b8f6c5bbd75e" (UID: "800631aa-8772-4844-865a-b8f6c5bbd75e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:44.842229 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.842191 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800631aa-8772-4844-865a-b8f6c5bbd75e-kube-api-access-9njqn" pod "800631aa-8772-4844-865a-b8f6c5bbd75e" (UID: "800631aa-8772-4844-865a-b8f6c5bbd75e"). InnerVolumeSpecName "kube-api-access-9njqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:44.845504 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.845473 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-metrics-certs" pod "800631aa-8772-4844-865a-b8f6c5bbd75e" (UID: "800631aa-8772-4844-865a-b8f6c5bbd75e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:44.940246 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.940208 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:44.940246 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.940237 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/800631aa-8772-4844-865a-b8f6c5bbd75e-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:44.940452 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.940315 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/800631aa-8772-4844-865a-b8f6c5bbd75e-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:44.940452 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.940324 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9njqn\" (UniqueName: \"kubernetes.io/projected/800631aa-8772-4844-865a-b8f6c5bbd75e-kube-api-access-9njqn\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:44.994693 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.994617 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4"] Nov 03 00:15:44.998303 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:44.998282 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-01450ec4-6b6c979d8c-hcks4"] Nov 03 00:15:45.115422 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:45.115393 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp"] Nov 03 00:15:45.124881 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:45.124852 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-01450ec4-5cf8bc545f-62cdp"] Nov 03 00:15:45.252798 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:45.252691 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9"] Nov 03 00:15:45.259203 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:45.259180 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01450ec4-6467967b79-rmzp9"] Nov 03 00:15:45.861102 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:45.861072 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw"] Nov 03 00:15:45.861488 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:45.861256 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" podUID="2fcf6502-70ee-4afc-a6ca-040130d60622" containerName="maas-api" containerID="cri-o://2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80" gracePeriod=30 Nov 03 00:15:46.101617 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.101591 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:15:46.252870 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.252831 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fcf6502-70ee-4afc-a6ca-040130d60622-kube-api-access-76mzc\" (UniqueName: \"kubernetes.io/projected/2fcf6502-70ee-4afc-a6ca-040130d60622-kube-api-access-76mzc\") pod \"2fcf6502-70ee-4afc-a6ca-040130d60622\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " Nov 03 00:15:46.253073 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.252891 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-metrics-certs\") pod \"2fcf6502-70ee-4afc-a6ca-040130d60622\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " Nov 03 00:15:46.253073 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.252939 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fcf6502-70ee-4afc-a6ca-040130d60622-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcf6502-70ee-4afc-a6ca-040130d60622-openshift-service-ca\") pod \"2fcf6502-70ee-4afc-a6ca-040130d60622\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " Nov 03 00:15:46.253073 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.252976 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-maas-api-tls\") pod \"2fcf6502-70ee-4afc-a6ca-040130d60622\" (UID: \"2fcf6502-70ee-4afc-a6ca-040130d60622\") " Nov 03 00:15:46.253402 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.253377 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fcf6502-70ee-4afc-a6ca-040130d60622-openshift-service-ca" pod "2fcf6502-70ee-4afc-a6ca-040130d60622" (UID: "2fcf6502-70ee-4afc-a6ca-040130d60622"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:46.255258 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.255233 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcf6502-70ee-4afc-a6ca-040130d60622-kube-api-access-76mzc" pod "2fcf6502-70ee-4afc-a6ca-040130d60622" (UID: "2fcf6502-70ee-4afc-a6ca-040130d60622"). InnerVolumeSpecName "kube-api-access-76mzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:46.255353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.255234 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-maas-api-tls" pod "2fcf6502-70ee-4afc-a6ca-040130d60622" (UID: "2fcf6502-70ee-4afc-a6ca-040130d60622"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:46.255353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.255260 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-metrics-certs" pod "2fcf6502-70ee-4afc-a6ca-040130d60622" (UID: "2fcf6502-70ee-4afc-a6ca-040130d60622"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:46.354697 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.354670 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:46.354697 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.354693 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76mzc\" (UniqueName: \"kubernetes.io/projected/2fcf6502-70ee-4afc-a6ca-040130d60622-kube-api-access-76mzc\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:46.354697 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.354703 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcf6502-70ee-4afc-a6ca-040130d60622-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:46.354928 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.354713 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcf6502-70ee-4afc-a6ca-040130d60622-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:46.771729 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.771672 4304 generic.go:356] "Generic (PLEG): container finished" podID="2fcf6502-70ee-4afc-a6ca-040130d60622" containerID="2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80" exitCode=0 Nov 03 00:15:46.771949 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.771742 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" Nov 03 00:15:46.771949 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.771756 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" event={"ID":"2fcf6502-70ee-4afc-a6ca-040130d60622","Type":"ContainerDied","Data":"2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80"} Nov 03 00:15:46.771949 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.771833 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw" event={"ID":"2fcf6502-70ee-4afc-a6ca-040130d60622","Type":"ContainerDied","Data":"497f404a3b0795ce5b0a0f18940eb98248d1eca9b228ced4dbff31e14647ca26"} Nov 03 00:15:46.771949 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.771859 4304 scope.go:122] "RemoveContainer" containerID="2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80" Nov 03 00:15:46.781616 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.781591 4304 scope.go:122] "RemoveContainer" containerID="2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80" Nov 03 00:15:46.781932 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:15:46.781913 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80\": container with ID starting with 2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80 not found: ID does not exist" containerID="2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80" Nov 03 00:15:46.781993 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.781940 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80"} err="failed to get container status \"2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80\": rpc error: code = NotFound desc = could not find container \"2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80\": container with ID starting with 2b3b2ae2f7607cabf8588f13024a083274ddf2fae45d0276a162518d3ae8cc80 not found: ID does not exist" Nov 03 00:15:46.835770 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.835739 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c4c48b-14af-44fa-8a6d-990972f28552" path="/var/lib/kubelet/pods/41c4c48b-14af-44fa-8a6d-990972f28552/volumes" Nov 03 00:15:46.836156 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.836143 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800631aa-8772-4844-865a-b8f6c5bbd75e" path="/var/lib/kubelet/pods/800631aa-8772-4844-865a-b8f6c5bbd75e/volumes" Nov 03 00:15:46.836453 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.836441 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c6af80-9613-473c-95c8-eb8c31922178" path="/var/lib/kubelet/pods/b5c6af80-9613-473c-95c8-eb8c31922178/volumes" Nov 03 00:15:46.939627 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.939594 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw"] Nov 03 00:15:46.990285 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:46.990254 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12537192-5d6cd95f4c-8hntw"] Nov 03 00:15:47.392334 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.392299 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7"] Nov 03 00:15:47.492030 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.491980 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" podUID="3757594c-a8fc-4fc1-b23d-a4d01b62f8e2" containerName="payload-processing" containerID="cri-o://3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462" gracePeriod=25 Nov 03 00:15:47.501292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.501264 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv"] Nov 03 00:15:47.602084 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.602043 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" podUID="bba22536-45fa-4f07-b43f-1a774948b0d7" containerName="payload-pre-processing" containerID="cri-o://6cab07dbf199c49deee55479aa143c880a136897b5b848ba61043e47b40cf606" gracePeriod=25 Nov 03 00:15:47.762543 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.762517 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:15:47.778647 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.778612 4304 generic.go:356] "Generic (PLEG): container finished" podID="bba22536-45fa-4f07-b43f-1a774948b0d7" containerID="6cab07dbf199c49deee55479aa143c880a136897b5b848ba61043e47b40cf606" exitCode=0 Nov 03 00:15:47.778818 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.778678 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" event={"ID":"bba22536-45fa-4f07-b43f-1a774948b0d7","Type":"ContainerDied","Data":"6cab07dbf199c49deee55479aa143c880a136897b5b848ba61043e47b40cf606"} Nov 03 00:15:47.781020 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.780997 4304 generic.go:356] "Generic (PLEG): container finished" podID="3757594c-a8fc-4fc1-b23d-a4d01b62f8e2" containerID="3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462" exitCode=0 Nov 03 00:15:47.781129 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.781036 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" Nov 03 00:15:47.781129 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.781073 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" event={"ID":"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2","Type":"ContainerDied","Data":"3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462"} Nov 03 00:15:47.781129 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.781121 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt" event={"ID":"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2","Type":"ContainerDied","Data":"57c6a0ed615971c9772717343bf61b5a3f7a8e41ffd331f801fc41daa0730b08"} Nov 03 00:15:47.781256 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.781144 4304 scope.go:122] "RemoveContainer" containerID="3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462" Nov 03 00:15:47.792896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.792877 4304 scope.go:122] "RemoveContainer" containerID="3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462" Nov 03 00:15:47.793212 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:15:47.793188 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462\": container with ID starting with 3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462 not found: ID does not exist" containerID="3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462" Nov 03 00:15:47.793295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.793221 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462"} err="failed to get container status \"3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462\": rpc error: code = NotFound desc = could not find container \"3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462\": container with ID starting with 3ae7afac40ec70fe5714f28bec123922dd5fa2a084991c09fd199bbea7678462 not found: ID does not exist" Nov 03 00:15:47.870034 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.870011 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-kube-api-access-2d2jd\" (UniqueName: \"kubernetes.io/projected/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-kube-api-access-2d2jd\") pod \"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2\" (UID: \"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2\") " Nov 03 00:15:47.870146 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.870069 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-plugins-config-volume\") pod \"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2\" (UID: \"3757594c-a8fc-4fc1-b23d-a4d01b62f8e2\") " Nov 03 00:15:47.870346 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.870326 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:15:47.870443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.870421 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-plugins-config-volume" pod "3757594c-a8fc-4fc1-b23d-a4d01b62f8e2" (UID: "3757594c-a8fc-4fc1-b23d-a4d01b62f8e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:47.872123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.872101 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-kube-api-access-2d2jd" pod "3757594c-a8fc-4fc1-b23d-a4d01b62f8e2" (UID: "3757594c-a8fc-4fc1-b23d-a4d01b62f8e2"). InnerVolumeSpecName "kube-api-access-2d2jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:47.971487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.971407 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba22536-45fa-4f07-b43f-1a774948b0d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba22536-45fa-4f07-b43f-1a774948b0d7-plugins-config-volume\") pod \"bba22536-45fa-4f07-b43f-1a774948b0d7\" (UID: \"bba22536-45fa-4f07-b43f-1a774948b0d7\") " Nov 03 00:15:47.971973 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.971488 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba22536-45fa-4f07-b43f-1a774948b0d7-kube-api-access-r8sch\" (UniqueName: \"kubernetes.io/projected/bba22536-45fa-4f07-b43f-1a774948b0d7-kube-api-access-r8sch\") pod \"bba22536-45fa-4f07-b43f-1a774948b0d7\" (UID: \"bba22536-45fa-4f07-b43f-1a774948b0d7\") " Nov 03 00:15:47.971973 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.971768 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d2jd\" (UniqueName: \"kubernetes.io/projected/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-kube-api-access-2d2jd\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:47.971973 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.971825 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:47.971973 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.971845 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba22536-45fa-4f07-b43f-1a774948b0d7-plugins-config-volume" pod "bba22536-45fa-4f07-b43f-1a774948b0d7" (UID: "bba22536-45fa-4f07-b43f-1a774948b0d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:47.973750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:47.973732 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba22536-45fa-4f07-b43f-1a774948b0d7-kube-api-access-r8sch" pod "bba22536-45fa-4f07-b43f-1a774948b0d7" (UID: "bba22536-45fa-4f07-b43f-1a774948b0d7"). InnerVolumeSpecName "kube-api-access-r8sch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:48.073158 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:48.073125 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba22536-45fa-4f07-b43f-1a774948b0d7-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:48.073158 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:48.073153 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8sch\" (UniqueName: \"kubernetes.io/projected/bba22536-45fa-4f07-b43f-1a774948b0d7-kube-api-access-r8sch\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:48.200799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:48.200735 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt"] Nov 03 00:15:48.208260 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:48.208232 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12537192-6f58444f67-q8spt"] Nov 03 00:15:48.787739 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:48.787670 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" Nov 03 00:15:48.787739 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:48.787697 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn" event={"ID":"bba22536-45fa-4f07-b43f-1a774948b0d7","Type":"ContainerDied","Data":"2894a3358ef895213ef946b895ea0c5ae8ad87520ea4633f7e1b4d215cfe7dcd"} Nov 03 00:15:48.787977 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:48.787766 4304 scope.go:122] "RemoveContainer" containerID="6cab07dbf199c49deee55479aa143c880a136897b5b848ba61043e47b40cf606" Nov 03 00:15:48.827608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:48.827578 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fcf6502-70ee-4afc-a6ca-040130d60622" path="/var/lib/kubelet/pods/2fcf6502-70ee-4afc-a6ca-040130d60622/volumes" Nov 03 00:15:48.827992 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:48.827978 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3757594c-a8fc-4fc1-b23d-a4d01b62f8e2" path="/var/lib/kubelet/pods/3757594c-a8fc-4fc1-b23d-a4d01b62f8e2/volumes" Nov 03 00:15:48.949659 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:48.949630 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn"] Nov 03 00:15:48.953894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:48.953864 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12537192-f6697cc47-hb5mn"] Nov 03 00:15:49.565252 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565218 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj"] Nov 03 00:15:49.565647 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565636 4304 cpu_manager.go:414] "containerMap: removing container" podUID="3757594c-a8fc-4fc1-b23d-a4d01b62f8e2" containerName="payload-processing" Nov 03 00:15:49.565689 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565647 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3757594c-a8fc-4fc1-b23d-a4d01b62f8e2" containerName="payload-processing" Nov 03 00:15:49.565689 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565673 4304 cpu_manager.go:414] "containerMap: removing container" podUID="41c4c48b-14af-44fa-8a6d-990972f28552" containerName="payload-processing" Nov 03 00:15:49.565689 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565677 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c4c48b-14af-44fa-8a6d-990972f28552" containerName="payload-processing" Nov 03 00:15:49.565689 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565684 4304 cpu_manager.go:414] "containerMap: removing container" podUID="2fcf6502-70ee-4afc-a6ca-040130d60622" containerName="maas-api" Nov 03 00:15:49.565689 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565689 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcf6502-70ee-4afc-a6ca-040130d60622" containerName="maas-api" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565700 4304 cpu_manager.go:414] "containerMap: removing container" podUID="800631aa-8772-4844-865a-b8f6c5bbd75e" containerName="maas-api" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565705 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800631aa-8772-4844-865a-b8f6c5bbd75e" containerName="maas-api" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565715 4304 cpu_manager.go:414] "containerMap: removing container" podUID="bba22536-45fa-4f07-b43f-1a774948b0d7" containerName="payload-pre-processing" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565720 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba22536-45fa-4f07-b43f-1a774948b0d7" containerName="payload-pre-processing" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565734 4304 cpu_manager.go:414] "containerMap: removing container" podUID="b5c6af80-9613-473c-95c8-eb8c31922178" containerName="payload-pre-processing" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565739 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c6af80-9613-473c-95c8-eb8c31922178" containerName="payload-pre-processing" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565807 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c4c48b-14af-44fa-8a6d-990972f28552" containerName="payload-processing" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565815 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcf6502-70ee-4afc-a6ca-040130d60622" containerName="maas-api" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565822 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="800631aa-8772-4844-865a-b8f6c5bbd75e" containerName="maas-api" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565827 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba22536-45fa-4f07-b43f-1a774948b0d7" containerName="payload-pre-processing" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565834 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c6af80-9613-473c-95c8-eb8c31922178" containerName="payload-pre-processing" Nov 03 00:15:49.565885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.565839 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3757594c-a8fc-4fc1-b23d-a4d01b62f8e2" containerName="payload-processing" Nov 03 00:15:49.570136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.570117 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.621660 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.621631 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-048135-openshift-default-dockercfg-xt942\"" Nov 03 00:15:49.673451 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.673416 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj"] Nov 03 00:15:49.685295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.685261 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.685417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.685308 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.685417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.685329 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.685417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.685353 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkbvb\" (UniqueName: \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-kube-api-access-xkbvb\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.685417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.685405 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.685563 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.685432 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.685563 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.685456 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.685563 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.685492 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.685563 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.685511 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.786878 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.786848 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.786878 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.786889 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.786909 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.786934 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkbvb\" (UniqueName: \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-kube-api-access-xkbvb\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.786993 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.787031 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.787057 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.787111 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.787137 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.787368 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.787406 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.787420 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787623 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.787500 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.787873 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.787849 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.789612 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.789588 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.789758 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.789737 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.848863 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.848756 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.848863 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.848852 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkbvb\" (UniqueName: \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-kube-api-access-xkbvb\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-9xfdj\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:49.882520 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:49.882486 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:50.176893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.176859 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4"] Nov 03 00:15:50.181847 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.181827 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.233762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.233736 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ebeeb9-openshift-default-dockercfg-pt5k5\"" Nov 03 00:15:50.243867 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.243836 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4"] Nov 03 00:15:50.292646 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.292616 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjlhw\" (UniqueName: \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-kube-api-access-zjlhw\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.292646 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.292648 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.292896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.292667 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.292896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.292746 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.292896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.292851 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.292896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.292892 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.293033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.292950 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.293033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.292973 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.293101 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.293047 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dabb012-d842-4c6b-8937-de5b400dbc46-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.337518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.337480 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj"] Nov 03 00:15:50.341731 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:15:50.341699 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31f9e36_ddf8_41a7_abcf_10d14702cb35.slice/crio-5b3de6b4de2e96f0ef8a025b9d1a968d5ae93123e3894920a9b3edfb44ca140c WatchSource:0}: Error finding container 5b3de6b4de2e96f0ef8a025b9d1a968d5ae93123e3894920a9b3edfb44ca140c: Status 404 returned error can't find the container with id 5b3de6b4de2e96f0ef8a025b9d1a968d5ae93123e3894920a9b3edfb44ca140c Nov 03 00:15:50.343969 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.343937 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:50.344059 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.344001 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:50.344059 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.344034 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:50.393525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.393500 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjlhw\" (UniqueName: \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-kube-api-access-zjlhw\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.393525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.393532 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.393719 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.393549 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.393719 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.393647 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.393719 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.393698 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.393916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.393764 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.393916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.393833 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.393916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.393861 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.394084 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.393940 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dabb012-d842-4c6b-8937-de5b400dbc46-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.394140 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.394076 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.394140 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.394107 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.394304 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.394288 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.394415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.394393 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.394598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.394571 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dabb012-d842-4c6b-8937-de5b400dbc46-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.395764 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.395746 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.396347 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.396332 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.450119 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.450029 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.450119 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.450079 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjlhw\" (UniqueName: \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-kube-api-access-zjlhw\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.495056 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.495029 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:50.802252 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.802168 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" event={"ID":"b31f9e36-ddf8-41a7-abcf-10d14702cb35","Type":"ContainerStarted","Data":"fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0"} Nov 03 00:15:50.802252 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.802219 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" event={"ID":"b31f9e36-ddf8-41a7-abcf-10d14702cb35","Type":"ContainerStarted","Data":"5b3de6b4de2e96f0ef8a025b9d1a968d5ae93123e3894920a9b3edfb44ca140c"} Nov 03 00:15:50.828371 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.828345 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba22536-45fa-4f07-b43f-1a774948b0d7" path="/var/lib/kubelet/pods/bba22536-45fa-4f07-b43f-1a774948b0d7/volumes" Nov 03 00:15:50.883561 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.883538 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:50.919305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.919280 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4"] Nov 03 00:15:50.923836 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.921193 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" podStartSLOduration=1.921173659 podStartE2EDuration="1.921173659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:50.919265049 +0000 UTC m=+634.202822724" watchObservedRunningTime="2026-11-03 00:15:50.921173659 +0000 UTC m=+634.204731336" Nov 03 00:15:50.923836 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:15:50.921928 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dabb012_d842_4c6b_8937_de5b400dbc46.slice/crio-74b5ea064c5c9636037a9cc07abaa3ee69216356bdbd19e3f9565884d21b8026 WatchSource:0}: Error finding container 74b5ea064c5c9636037a9cc07abaa3ee69216356bdbd19e3f9565884d21b8026: Status 404 returned error can't find the container with id 74b5ea064c5c9636037a9cc07abaa3ee69216356bdbd19e3f9565884d21b8026 Nov 03 00:15:50.924580 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.924545 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:50.924649 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.924621 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:50.924706 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:50.924659 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:51.808988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:51.808948 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" event={"ID":"6dabb012-d842-4c6b-8937-de5b400dbc46","Type":"ContainerStarted","Data":"27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9"} Nov 03 00:15:51.809404 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:51.809012 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" event={"ID":"6dabb012-d842-4c6b-8937-de5b400dbc46","Type":"ContainerStarted","Data":"74b5ea064c5c9636037a9cc07abaa3ee69216356bdbd19e3f9565884d21b8026"} Nov 03 00:15:51.888307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:51.888280 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:51.928014 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:51.927964 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" podStartSLOduration=1.927951371 podStartE2EDuration="1.927951371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:51.927459968 +0000 UTC m=+635.211017642" watchObservedRunningTime="2026-11-03 00:15:51.927951371 +0000 UTC m=+635.211509045" Nov 03 00:15:52.392841 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.392752 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" podUID="ade6cd92-b99f-4799-b73d-878bece393d9" containerName="payload-processing" containerID="cri-o://5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6" gracePeriod=25 Nov 03 00:15:52.495747 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.495709 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:52.501075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.501050 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:52.502027 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.501998 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" podUID="289daeb7-1acb-4aad-a229-af361e633761" containerName="payload-pre-processing" containerID="cri-o://44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011" gracePeriod=25 Nov 03 00:15:52.647590 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.647523 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:15:52.716057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.716024 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade6cd92-b99f-4799-b73d-878bece393d9-kube-api-access-blpr8\" (UniqueName: \"kubernetes.io/projected/ade6cd92-b99f-4799-b73d-878bece393d9-kube-api-access-blpr8\") pod \"ade6cd92-b99f-4799-b73d-878bece393d9\" (UID: \"ade6cd92-b99f-4799-b73d-878bece393d9\") " Nov 03 00:15:52.716224 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.716160 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade6cd92-b99f-4799-b73d-878bece393d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade6cd92-b99f-4799-b73d-878bece393d9-plugins-config-volume\") pod \"ade6cd92-b99f-4799-b73d-878bece393d9\" (UID: \"ade6cd92-b99f-4799-b73d-878bece393d9\") " Nov 03 00:15:52.716997 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.716965 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade6cd92-b99f-4799-b73d-878bece393d9-plugins-config-volume" pod "ade6cd92-b99f-4799-b73d-878bece393d9" (UID: "ade6cd92-b99f-4799-b73d-878bece393d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:52.719343 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.719311 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade6cd92-b99f-4799-b73d-878bece393d9-kube-api-access-blpr8" pod "ade6cd92-b99f-4799-b73d-878bece393d9" (UID: "ade6cd92-b99f-4799-b73d-878bece393d9"). InnerVolumeSpecName "kube-api-access-blpr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:52.748972 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.748953 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:15:52.816509 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.816481 4304 generic.go:356] "Generic (PLEG): container finished" podID="ade6cd92-b99f-4799-b73d-878bece393d9" containerID="5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6" exitCode=0 Nov 03 00:15:52.816998 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.816559 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" event={"ID":"ade6cd92-b99f-4799-b73d-878bece393d9","Type":"ContainerDied","Data":"5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6"} Nov 03 00:15:52.816998 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.816609 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" event={"ID":"ade6cd92-b99f-4799-b73d-878bece393d9","Type":"ContainerDied","Data":"ca67ed23f49886b9bcf665be259dfb67db5e64dba70e6bcb81f1ed6474871082"} Nov 03 00:15:52.816998 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.816632 4304 scope.go:122] "RemoveContainer" containerID="5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6" Nov 03 00:15:52.816998 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.816561 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7" Nov 03 00:15:52.816998 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.816695 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289daeb7-1acb-4aad-a229-af361e633761-kube-api-access-7qpn9\" (UniqueName: \"kubernetes.io/projected/289daeb7-1acb-4aad-a229-af361e633761-kube-api-access-7qpn9\") pod \"289daeb7-1acb-4aad-a229-af361e633761\" (UID: \"289daeb7-1acb-4aad-a229-af361e633761\") " Nov 03 00:15:52.816998 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.816882 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289daeb7-1acb-4aad-a229-af361e633761-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289daeb7-1acb-4aad-a229-af361e633761-plugins-config-volume\") pod \"289daeb7-1acb-4aad-a229-af361e633761\" (UID: \"289daeb7-1acb-4aad-a229-af361e633761\") " Nov 03 00:15:52.817309 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.817164 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade6cd92-b99f-4799-b73d-878bece393d9-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:52.817309 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.817186 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blpr8\" (UniqueName: \"kubernetes.io/projected/ade6cd92-b99f-4799-b73d-878bece393d9-kube-api-access-blpr8\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:52.817380 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.817330 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289daeb7-1acb-4aad-a229-af361e633761-plugins-config-volume" pod "289daeb7-1acb-4aad-a229-af361e633761" (UID: "289daeb7-1acb-4aad-a229-af361e633761"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:52.818821 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.818765 4304 generic.go:356] "Generic (PLEG): container finished" podID="289daeb7-1acb-4aad-a229-af361e633761" containerID="44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011" exitCode=0 Nov 03 00:15:52.818948 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.818903 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" Nov 03 00:15:52.818998 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.818977 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" event={"ID":"289daeb7-1acb-4aad-a229-af361e633761","Type":"ContainerDied","Data":"44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011"} Nov 03 00:15:52.819058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.819019 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv" event={"ID":"289daeb7-1acb-4aad-a229-af361e633761","Type":"ContainerDied","Data":"8e46e7992b78f6e5f16a7b9552ef2358def0c30f7a1d0ac1a1f4510a806a74fc"} Nov 03 00:15:52.819512 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.819431 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:52.819512 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.819481 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:52.819512 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.819500 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289daeb7-1acb-4aad-a229-af361e633761-kube-api-access-7qpn9" pod "289daeb7-1acb-4aad-a229-af361e633761" (UID: "289daeb7-1acb-4aad-a229-af361e633761"). InnerVolumeSpecName "kube-api-access-7qpn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:52.820281 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.820258 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:15:52.820410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.820392 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:15:52.827346 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.827326 4304 scope.go:122] "RemoveContainer" containerID="5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6" Nov 03 00:15:52.827645 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:15:52.827626 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6\": container with ID starting with 5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6 not found: ID does not exist" containerID="5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6" Nov 03 00:15:52.827692 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.827653 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6"} err="failed to get container status \"5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6\": rpc error: code = NotFound desc = could not find container \"5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6\": container with ID starting with 5e5c46088d725cbe12e8a3bd715e72510d7957361b60b8a4f1f91ebad853e4c6 not found: ID does not exist" Nov 03 00:15:52.827692 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.827670 4304 scope.go:122] "RemoveContainer" containerID="44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011" Nov 03 00:15:52.835996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.835977 4304 scope.go:122] "RemoveContainer" containerID="44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011" Nov 03 00:15:52.836263 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:15:52.836245 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011\": container with ID starting with 44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011 not found: ID does not exist" containerID="44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011" Nov 03 00:15:52.836317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.836268 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011"} err="failed to get container status \"44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011\": rpc error: code = NotFound desc = could not find container \"44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011\": container with ID starting with 44d33eeea7d759d73d84fe44f58106bfdba45eeaa86fb805d3f7d1486789b011 not found: ID does not exist" Nov 03 00:15:52.918641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.918562 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qpn9\" (UniqueName: \"kubernetes.io/projected/289daeb7-1acb-4aad-a229-af361e633761-kube-api-access-7qpn9\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:52.918641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:52.918617 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289daeb7-1acb-4aad-a229-af361e633761-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:53.219453 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:53.219378 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7"] Nov 03 00:15:53.227157 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:53.226900 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ccce380-7cbdbf5849-7lmd7"] Nov 03 00:15:53.385700 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:53.385663 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv"] Nov 03 00:15:53.391246 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:53.391218 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ccce380-79f6dcf559-gbthv"] Nov 03 00:15:53.966585 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:53.966556 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c"] Nov 03 00:15:54.829117 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:54.829088 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289daeb7-1acb-4aad-a229-af361e633761" path="/var/lib/kubelet/pods/289daeb7-1acb-4aad-a229-af361e633761/volumes" Nov 03 00:15:54.829600 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:54.829489 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade6cd92-b99f-4799-b73d-878bece393d9" path="/var/lib/kubelet/pods/ade6cd92-b99f-4799-b73d-878bece393d9/volumes" Nov 03 00:15:57.646645 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:57.646608 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk"] Nov 03 00:15:57.647157 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:57.646824 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" podUID="80fd667f-313f-429c-95e1-a8856cca4122" containerName="maas-api" containerID="cri-o://e885a597619b60fc97ec96376ec957da7789ba7d8adbfb6356ad247f751273ef" gracePeriod=30 Nov 03 00:15:57.850326 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:57.850265 4304 generic.go:356] "Generic (PLEG): container finished" podID="80fd667f-313f-429c-95e1-a8856cca4122" containerID="e885a597619b60fc97ec96376ec957da7789ba7d8adbfb6356ad247f751273ef" exitCode=0 Nov 03 00:15:57.850580 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:57.850404 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" event={"ID":"80fd667f-313f-429c-95e1-a8856cca4122","Type":"ContainerDied","Data":"e885a597619b60fc97ec96376ec957da7789ba7d8adbfb6356ad247f751273ef"} Nov 03 00:15:57.911965 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:57.911942 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:58.080691 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.080664 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80fd667f-313f-429c-95e1-a8856cca4122-kube-api-access-65d56\" (UniqueName: \"kubernetes.io/projected/80fd667f-313f-429c-95e1-a8856cca4122-kube-api-access-65d56\") pod \"80fd667f-313f-429c-95e1-a8856cca4122\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " Nov 03 00:15:58.080868 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.080704 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-metrics-certs\") pod \"80fd667f-313f-429c-95e1-a8856cca4122\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " Nov 03 00:15:58.080868 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.080730 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-maas-api-tls\") pod \"80fd667f-313f-429c-95e1-a8856cca4122\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " Nov 03 00:15:58.081020 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.080901 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80fd667f-313f-429c-95e1-a8856cca4122-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80fd667f-313f-429c-95e1-a8856cca4122-openshift-service-ca\") pod \"80fd667f-313f-429c-95e1-a8856cca4122\" (UID: \"80fd667f-313f-429c-95e1-a8856cca4122\") " Nov 03 00:15:58.081244 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.081220 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fd667f-313f-429c-95e1-a8856cca4122-openshift-service-ca" pod "80fd667f-313f-429c-95e1-a8856cca4122" (UID: "80fd667f-313f-429c-95e1-a8856cca4122"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:58.081396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.081380 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80fd667f-313f-429c-95e1-a8856cca4122-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:58.082921 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.082890 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fd667f-313f-429c-95e1-a8856cca4122-kube-api-access-65d56" pod "80fd667f-313f-429c-95e1-a8856cca4122" (UID: "80fd667f-313f-429c-95e1-a8856cca4122"). InnerVolumeSpecName "kube-api-access-65d56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:58.082921 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.082892 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-maas-api-tls" pod "80fd667f-313f-429c-95e1-a8856cca4122" (UID: "80fd667f-313f-429c-95e1-a8856cca4122"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:58.083055 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.082980 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-metrics-certs" pod "80fd667f-313f-429c-95e1-a8856cca4122" (UID: "80fd667f-313f-429c-95e1-a8856cca4122"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:58.182236 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.182159 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65d56\" (UniqueName: \"kubernetes.io/projected/80fd667f-313f-429c-95e1-a8856cca4122-kube-api-access-65d56\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:58.182236 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.182189 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:58.182236 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.182199 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80fd667f-313f-429c-95e1-a8856cca4122-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:58.654885 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:15:58.654846 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:15:58.856222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.856198 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" Nov 03 00:15:58.856363 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.856229 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk" event={"ID":"80fd667f-313f-429c-95e1-a8856cca4122","Type":"ContainerDied","Data":"70157279089f2944a1a6392f3ee47356c7d48f9d35a5c75036ef82b1f5065656"} Nov 03 00:15:58.856363 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.856288 4304 scope.go:122] "RemoveContainer" containerID="e885a597619b60fc97ec96376ec957da7789ba7d8adbfb6356ad247f751273ef" Nov 03 00:15:58.967711 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:58.967621 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" podUID="5a591f05-0c35-4309-a2e2-f43e898c6918" containerName="payload-processing" containerID="cri-o://3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b" gracePeriod=25 Nov 03 00:15:59.018906 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.018879 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk"] Nov 03 00:15:59.069732 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.069702 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-168f474d-64c86d64d6-bscrk"] Nov 03 00:15:59.217874 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.217816 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:59.391733 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.391702 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a591f05-0c35-4309-a2e2-f43e898c6918-kube-api-access-mv4sg\" (UniqueName: \"kubernetes.io/projected/5a591f05-0c35-4309-a2e2-f43e898c6918-kube-api-access-mv4sg\") pod \"5a591f05-0c35-4309-a2e2-f43e898c6918\" (UID: \"5a591f05-0c35-4309-a2e2-f43e898c6918\") " Nov 03 00:15:59.391923 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.391867 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a591f05-0c35-4309-a2e2-f43e898c6918-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a591f05-0c35-4309-a2e2-f43e898c6918-plugins-config-volume\") pod \"5a591f05-0c35-4309-a2e2-f43e898c6918\" (UID: \"5a591f05-0c35-4309-a2e2-f43e898c6918\") " Nov 03 00:15:59.392214 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.392191 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a591f05-0c35-4309-a2e2-f43e898c6918-plugins-config-volume" pod "5a591f05-0c35-4309-a2e2-f43e898c6918" (UID: "5a591f05-0c35-4309-a2e2-f43e898c6918"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:59.394223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.394203 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a591f05-0c35-4309-a2e2-f43e898c6918-kube-api-access-mv4sg" pod "5a591f05-0c35-4309-a2e2-f43e898c6918" (UID: "5a591f05-0c35-4309-a2e2-f43e898c6918"). InnerVolumeSpecName "kube-api-access-mv4sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:59.493208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.493133 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a591f05-0c35-4309-a2e2-f43e898c6918-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:59.493208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.493159 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv4sg\" (UniqueName: \"kubernetes.io/projected/5a591f05-0c35-4309-a2e2-f43e898c6918-kube-api-access-mv4sg\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:59.862842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.862739 4304 generic.go:356] "Generic (PLEG): container finished" podID="5a591f05-0c35-4309-a2e2-f43e898c6918" containerID="3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b" exitCode=0 Nov 03 00:15:59.862842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.862807 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" Nov 03 00:15:59.862842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.862806 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" event={"ID":"5a591f05-0c35-4309-a2e2-f43e898c6918","Type":"ContainerDied","Data":"3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b"} Nov 03 00:15:59.863343 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.862867 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c" event={"ID":"5a591f05-0c35-4309-a2e2-f43e898c6918","Type":"ContainerDied","Data":"fd9aa3b4e68d97b2867027b77a8986b4953b444a9c71b49a3041532e00141ce3"} Nov 03 00:15:59.863343 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.862895 4304 scope.go:122] "RemoveContainer" containerID="3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b" Nov 03 00:15:59.872231 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.872217 4304 scope.go:122] "RemoveContainer" containerID="3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b" Nov 03 00:15:59.872487 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:15:59.872469 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b\": container with ID starting with 3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b not found: ID does not exist" containerID="3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b" Nov 03 00:15:59.872529 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:15:59.872496 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b"} err="failed to get container status \"3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b\": rpc error: code = NotFound desc = could not find container \"3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b\": container with ID starting with 3cdcb6ffbd1b6d45b52ec48169076b207a5d76f53757f9abb43e8a7169b2fb3b not found: ID does not exist" Nov 03 00:16:00.029199 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.029169 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c"] Nov 03 00:16:00.078642 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.078611 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-168f474d-84c7578d75-bxv4c"] Nov 03 00:16:00.495513 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.495481 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz"] Nov 03 00:16:00.495741 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.495717 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" containerID="cri-o://b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba" gracePeriod=30 Nov 03 00:16:00.828914 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.828835 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a591f05-0c35-4309-a2e2-f43e898c6918" path="/var/lib/kubelet/pods/5a591f05-0c35-4309-a2e2-f43e898c6918/volumes" Nov 03 00:16:00.829246 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.829233 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80fd667f-313f-429c-95e1-a8856cca4122" path="/var/lib/kubelet/pods/80fd667f-313f-429c-95e1-a8856cca4122/volumes" Nov 03 00:16:00.896597 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.896569 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6"] Nov 03 00:16:00.896997 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.896988 4304 cpu_manager.go:414] "containerMap: removing container" podUID="289daeb7-1acb-4aad-a229-af361e633761" containerName="payload-pre-processing" Nov 03 00:16:00.897033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.897002 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289daeb7-1acb-4aad-a229-af361e633761" containerName="payload-pre-processing" Nov 03 00:16:00.897033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.897013 4304 cpu_manager.go:414] "containerMap: removing container" podUID="ade6cd92-b99f-4799-b73d-878bece393d9" containerName="payload-processing" Nov 03 00:16:00.897033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.897018 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade6cd92-b99f-4799-b73d-878bece393d9" containerName="payload-processing" Nov 03 00:16:00.897137 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.897035 4304 cpu_manager.go:414] "containerMap: removing container" podUID="80fd667f-313f-429c-95e1-a8856cca4122" containerName="maas-api" Nov 03 00:16:00.897137 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.897040 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80fd667f-313f-429c-95e1-a8856cca4122" containerName="maas-api" Nov 03 00:16:00.897137 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.897052 4304 cpu_manager.go:414] "containerMap: removing container" podUID="5a591f05-0c35-4309-a2e2-f43e898c6918" containerName="payload-processing" Nov 03 00:16:00.897137 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.897057 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a591f05-0c35-4309-a2e2-f43e898c6918" containerName="payload-processing" Nov 03 00:16:00.897137 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.897110 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="289daeb7-1acb-4aad-a229-af361e633761" containerName="payload-pre-processing" Nov 03 00:16:00.897137 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.897118 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade6cd92-b99f-4799-b73d-878bece393d9" containerName="payload-processing" Nov 03 00:16:00.897137 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.897128 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="80fd667f-313f-429c-95e1-a8856cca4122" containerName="maas-api" Nov 03 00:16:00.897137 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.897133 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a591f05-0c35-4309-a2e2-f43e898c6918" containerName="payload-processing" Nov 03 00:16:00.901316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.901297 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 00:16:00.908334 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.908311 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8x9s\" (UniqueName: \"kubernetes.io/projected/010ce23a-6dab-4d92-a5ce-acc072a62fd0-kube-api-access-l8x9s\") pod \"payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6\" (UID: \"010ce23a-6dab-4d92-a5ce-acc072a62fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 00:16:00.908428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.908410 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010ce23a-6dab-4d92-a5ce-acc072a62fd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6\" (UID: \"010ce23a-6dab-4d92-a5ce-acc072a62fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 00:16:00.956233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.956186 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ebeeb9\"" Nov 03 00:16:00.956471 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.956450 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ebeeb9-dockercfg-mpndb\"" Nov 03 00:16:00.963239 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:00.963208 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6"] Nov 03 00:16:01.009007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.008973 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010ce23a-6dab-4d92-a5ce-acc072a62fd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6\" (UID: \"010ce23a-6dab-4d92-a5ce-acc072a62fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 00:16:01.009247 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.009099 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8x9s\" (UniqueName: \"kubernetes.io/projected/010ce23a-6dab-4d92-a5ce-acc072a62fd0-kube-api-access-l8x9s\") pod \"payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6\" (UID: \"010ce23a-6dab-4d92-a5ce-acc072a62fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 00:16:01.009920 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.009901 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010ce23a-6dab-4d92-a5ce-acc072a62fd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6\" (UID: \"010ce23a-6dab-4d92-a5ce-acc072a62fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 00:16:01.023583 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.023552 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm"] Nov 03 00:16:01.027100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.027084 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 00:16:01.066923 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.066898 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8x9s\" (UniqueName: \"kubernetes.io/projected/010ce23a-6dab-4d92-a5ce-acc072a62fd0-kube-api-access-l8x9s\") pod \"payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6\" (UID: \"010ce23a-6dab-4d92-a5ce-acc072a62fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 00:16:01.085553 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.085491 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm"] Nov 03 00:16:01.109597 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.109567 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmlm5\" (UniqueName: \"kubernetes.io/projected/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-kube-api-access-hmlm5\") pod \"payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm\" (UID: \"4d155e9a-254a-4bb0-a535-1c237d7e1f8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 00:16:01.109765 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.109619 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm\" (UID: \"4d155e9a-254a-4bb0-a535-1c237d7e1f8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 00:16:01.210578 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.210548 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm\" (UID: \"4d155e9a-254a-4bb0-a535-1c237d7e1f8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 00:16:01.210724 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.210650 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmlm5\" (UniqueName: \"kubernetes.io/projected/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-kube-api-access-hmlm5\") pod \"payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm\" (UID: \"4d155e9a-254a-4bb0-a535-1c237d7e1f8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 00:16:01.211566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.211542 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm\" (UID: \"4d155e9a-254a-4bb0-a535-1c237d7e1f8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 00:16:01.213417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.213398 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 00:16:01.267762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.267584 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmlm5\" (UniqueName: \"kubernetes.io/projected/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-kube-api-access-hmlm5\") pod \"payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm\" (UID: \"4d155e9a-254a-4bb0-a535-1c237d7e1f8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 00:16:01.340829 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.340734 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 00:16:01.606698 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.606677 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6"] Nov 03 00:16:01.607768 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:16:01.607738 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010ce23a_6dab_4d92_a5ce_acc072a62fd0.slice/crio-cefc693f03b0de4c1b630735c09051897117aff1309a36d4b7b51b38164212f1 WatchSource:0}: Error finding container cefc693f03b0de4c1b630735c09051897117aff1309a36d4b7b51b38164212f1: Status 404 returned error can't find the container with id cefc693f03b0de4c1b630735c09051897117aff1309a36d4b7b51b38164212f1 Nov 03 00:16:01.634828 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.634803 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp"] Nov 03 00:16:01.718555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.718530 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm"] Nov 03 00:16:01.721579 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:16:01.721541 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d155e9a_254a_4bb0_a535_1c237d7e1f8c.slice/crio-a85873914764229b05fc0b62848c7b9663473f7a2884d7fe59f20352e9a86a1e WatchSource:0}: Error finding container a85873914764229b05fc0b62848c7b9663473f7a2884d7fe59f20352e9a86a1e: Status 404 returned error can't find the container with id a85873914764229b05fc0b62848c7b9663473f7a2884d7fe59f20352e9a86a1e Nov 03 00:16:01.724313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.724293 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6"] Nov 03 00:16:01.886192 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.886076 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" event={"ID":"4d155e9a-254a-4bb0-a535-1c237d7e1f8c","Type":"ContainerStarted","Data":"9ba6608df492dc5dbe1b40cfa7c7a2a93bedfc56fc231b341aec9b14fe7218b7"} Nov 03 00:16:01.886504 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.886467 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 00:16:01.886649 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.886515 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" event={"ID":"4d155e9a-254a-4bb0-a535-1c237d7e1f8c","Type":"ContainerStarted","Data":"a85873914764229b05fc0b62848c7b9663473f7a2884d7fe59f20352e9a86a1e"} Nov 03 00:16:01.888807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.888744 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" event={"ID":"010ce23a-6dab-4d92-a5ce-acc072a62fd0","Type":"ContainerStarted","Data":"7ae23cd759b45dcd8fd30ede9e72ab88d4e33e2b3be363440822acfffad3af84"} Nov 03 00:16:01.889017 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.888986 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" event={"ID":"010ce23a-6dab-4d92-a5ce-acc072a62fd0","Type":"ContainerStarted","Data":"cefc693f03b0de4c1b630735c09051897117aff1309a36d4b7b51b38164212f1"} Nov 03 00:16:01.889223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:01.889203 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 00:16:02.010222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:02.010157 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" podStartSLOduration=2.01014068 podStartE2EDuration="2.01014068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:02.009888972 +0000 UTC m=+645.293446648" watchObservedRunningTime="2026-11-03 00:16:02.01014068 +0000 UTC m=+645.293698355" Nov 03 00:16:02.130227 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:02.130151 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" podStartSLOduration=2.130126082 podStartE2EDuration="2.130126082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:02.127229304 +0000 UTC m=+645.410786980" watchObservedRunningTime="2026-11-03 00:16:02.130126082 +0000 UTC m=+645.413683757" Nov 03 00:16:02.152392 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:02.152347 4304 patch_prober.go:29] interesting pod/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:16:02.152547 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:02.152411 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:16:05.258617 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.258583 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc"] Nov 03 00:16:05.262428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.262408 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.313949 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.313927 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-048135-serving-cert\"" Nov 03 00:16:05.347621 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.347591 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c95304-1ac6-4047-ad06-09946b5161dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.347749 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.347638 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-metrics-certs\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.347825 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.347762 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mqqm\" (UniqueName: \"kubernetes.io/projected/19c95304-1ac6-4047-ad06-09946b5161dd-kube-api-access-8mqqm\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.347984 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.347962 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-maas-api-tls\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.365983 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.365955 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc"] Nov 03 00:16:05.449352 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.449321 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-maas-api-tls\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.449534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.449406 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c95304-1ac6-4047-ad06-09946b5161dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.449534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.449435 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-metrics-certs\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.449534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.449465 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mqqm\" (UniqueName: \"kubernetes.io/projected/19c95304-1ac6-4047-ad06-09946b5161dd-kube-api-access-8mqqm\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.449534 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:16:05.449473 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-serving-cert: secret "maas-api-e2e-worker-w4-048135-serving-cert" not found Nov 03 00:16:05.449715 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:16:05.449546 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-maas-api-tls podName:19c95304-1ac6-4047-ad06-09946b5161dd nodeName:}" failed. No retries permitted until 2026-11-03 00:16:05.949525684 +0000 UTC m=+649.233083337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-maas-api-tls") pod "maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" (UID: "19c95304-1ac6-4047-ad06-09946b5161dd") : secret "maas-api-e2e-worker-w4-048135-serving-cert" not found Nov 03 00:16:05.450348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.450322 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c95304-1ac6-4047-ad06-09946b5161dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.452133 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.452109 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-metrics-certs\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.519912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.519889 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mqqm\" (UniqueName: \"kubernetes.io/projected/19c95304-1ac6-4047-ad06-09946b5161dd-kube-api-access-8mqqm\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.749220 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.749193 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:16:05.854007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.853927 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-token\" (UniqueName: \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-token\") pod \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " Nov 03 00:16:05.854007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.853970 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-data\") pod \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " Nov 03 00:16:05.854007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.853990 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-kube-api-access-rbh4j\" (UniqueName: \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-kube-api-access-rbh4j\") pod \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " Nov 03 00:16:05.854264 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.854080 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-envoy\") pod \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " Nov 03 00:16:05.854264 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.854132 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-certs\") pod \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " Nov 03 00:16:05.854264 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.854171 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istiod-ca-cert\") pod \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " Nov 03 00:16:05.854264 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.854210 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-podinfo\") pod \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " Nov 03 00:16:05.854264 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.854247 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-socket\") pod \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " Nov 03 00:16:05.854264 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.854252 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-data" pod "135c8d1b-e9c2-4f5a-8b05-ea277767f83b" (UID: "135c8d1b-e9c2-4f5a-8b05-ea277767f83b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:05.854569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.854293 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-credential-socket\") pod \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\" (UID: \"135c8d1b-e9c2-4f5a-8b05-ea277767f83b\") " Nov 03 00:16:05.854569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.854448 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-certs" pod "135c8d1b-e9c2-4f5a-8b05-ea277767f83b" (UID: "135c8d1b-e9c2-4f5a-8b05-ea277767f83b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:05.854569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.854559 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-socket" pod "135c8d1b-e9c2-4f5a-8b05-ea277767f83b" (UID: "135c8d1b-e9c2-4f5a-8b05-ea277767f83b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:05.855282 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.854807 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istiod-ca-cert" pod "135c8d1b-e9c2-4f5a-8b05-ea277767f83b" (UID: "135c8d1b-e9c2-4f5a-8b05-ea277767f83b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:05.855282 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.854814 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-credential-socket" pod "135c8d1b-e9c2-4f5a-8b05-ea277767f83b" (UID: "135c8d1b-e9c2-4f5a-8b05-ea277767f83b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:05.855482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.855298 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:05.855482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.855414 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:05.855482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.855435 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:05.855482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.855449 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:05.855482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.855466 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:05.856554 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.856529 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-podinfo" pod "135c8d1b-e9c2-4f5a-8b05-ea277767f83b" (UID: "135c8d1b-e9c2-4f5a-8b05-ea277767f83b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:16:05.856648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.856552 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-token" pod "135c8d1b-e9c2-4f5a-8b05-ea277767f83b" (UID: "135c8d1b-e9c2-4f5a-8b05-ea277767f83b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:05.856648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.856564 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-kube-api-access-rbh4j" pod "135c8d1b-e9c2-4f5a-8b05-ea277767f83b" (UID: "135c8d1b-e9c2-4f5a-8b05-ea277767f83b"). InnerVolumeSpecName "kube-api-access-rbh4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:05.856648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.856593 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-envoy" pod "135c8d1b-e9c2-4f5a-8b05-ea277767f83b" (UID: "135c8d1b-e9c2-4f5a-8b05-ea277767f83b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:05.910832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.910767 4304 generic.go:356] "Generic (PLEG): container finished" podID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerID="b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba" exitCode=0 Nov 03 00:16:05.910996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.910927 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" event={"ID":"135c8d1b-e9c2-4f5a-8b05-ea277767f83b","Type":"ContainerDied","Data":"b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba"} Nov 03 00:16:05.910996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.910967 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" event={"ID":"135c8d1b-e9c2-4f5a-8b05-ea277767f83b","Type":"ContainerDied","Data":"21fbed81b8a35c773ee079b7cb566cf754ecd5cfbff676da915112578b32985f"} Nov 03 00:16:05.910996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.910990 4304 scope.go:122] "RemoveContainer" containerID="b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba" Nov 03 00:16:05.911159 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.911127 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz" Nov 03 00:16:05.926527 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.926500 4304 scope.go:122] "RemoveContainer" containerID="591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf" Nov 03 00:16:05.936848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.936826 4304 scope.go:122] "RemoveContainer" containerID="b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba" Nov 03 00:16:05.937157 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:16:05.937127 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba\": container with ID starting with b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba not found: ID does not exist" containerID="b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba" Nov 03 00:16:05.937240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.937167 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba"} err="failed to get container status \"b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba\": rpc error: code = NotFound desc = could not find container \"b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba\": container with ID starting with b4e02faf9d779a4ca54d887a60d8bd183e37ad60d0296aad119d101a5ddae3ba not found: ID does not exist" Nov 03 00:16:05.937240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.937186 4304 scope.go:122] "RemoveContainer" containerID="591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf" Nov 03 00:16:05.937472 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:16:05.937444 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf\": container with ID starting with 591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf not found: ID does not exist" containerID="591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf" Nov 03 00:16:05.937573 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.937479 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf"} err="failed to get container status \"591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf\": rpc error: code = NotFound desc = could not find container \"591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf\": container with ID starting with 591ff67e02e25bdfa0fb871fc94b9a657fdc8b4e81a7762740f012fd840736cf not found: ID does not exist" Nov 03 00:16:05.956032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.956009 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-maas-api-tls\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:05.956173 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.956107 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:05.956173 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.956123 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:05.956173 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.956137 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:05.956173 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.956150 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbh4j\" (UniqueName: \"kubernetes.io/projected/135c8d1b-e9c2-4f5a-8b05-ea277767f83b-kube-api-access-rbh4j\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:05.958396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:05.958378 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-maas-api-tls\") pod \"maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:06.078119 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.078082 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz"] Nov 03 00:16:06.081832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.081753 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-168f474d-openshift-default-745957bd8f-rs6bz"] Nov 03 00:16:06.175544 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.175515 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:06.552843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.552816 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc"] Nov 03 00:16:06.554477 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:16:06.554449 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c95304_1ac6_4047_ad06_09946b5161dd.slice/crio-246171495f9380fd3db854ed0d9a753f169c25b08328d0842ed5f4f976c96649 WatchSource:0}: Error finding container 246171495f9380fd3db854ed0d9a753f169c25b08328d0842ed5f4f976c96649: Status 404 returned error can't find the container with id 246171495f9380fd3db854ed0d9a753f169c25b08328d0842ed5f4f976c96649 Nov 03 00:16:06.555959 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.555940 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:16:06.635869 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.635812 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" podUID="bc16e901-e277-46c6-96f7-2cf983dd9e03" containerName="payload-processing" containerID="cri-o://32ac2f4381308c55934091d26fd0657d9afd703547e025190d87c9b3c5247688" gracePeriod=25 Nov 03 00:16:06.724618 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.724526 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" podUID="8e31dcbe-63d2-49b9-84da-ee9d0914d377" containerName="payload-pre-processing" containerID="cri-o://271453b76f0e795f8bfd8d00b03bdecc127b7f89283e1bc9ad279904af29aa8f" gracePeriod=25 Nov 03 00:16:06.830361 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.830338 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" path="/var/lib/kubelet/pods/135c8d1b-e9c2-4f5a-8b05-ea277767f83b/volumes" Nov 03 00:16:06.919680 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.919650 4304 generic.go:356] "Generic (PLEG): container finished" podID="bc16e901-e277-46c6-96f7-2cf983dd9e03" containerID="32ac2f4381308c55934091d26fd0657d9afd703547e025190d87c9b3c5247688" exitCode=0 Nov 03 00:16:06.919866 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.919730 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" event={"ID":"bc16e901-e277-46c6-96f7-2cf983dd9e03","Type":"ContainerDied","Data":"32ac2f4381308c55934091d26fd0657d9afd703547e025190d87c9b3c5247688"} Nov 03 00:16:06.921266 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.921223 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" event={"ID":"19c95304-1ac6-4047-ad06-09946b5161dd","Type":"ContainerStarted","Data":"246171495f9380fd3db854ed0d9a753f169c25b08328d0842ed5f4f976c96649"} Nov 03 00:16:06.923141 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.923118 4304 generic.go:356] "Generic (PLEG): container finished" podID="8e31dcbe-63d2-49b9-84da-ee9d0914d377" containerID="271453b76f0e795f8bfd8d00b03bdecc127b7f89283e1bc9ad279904af29aa8f" exitCode=0 Nov 03 00:16:06.923261 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.923185 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" event={"ID":"8e31dcbe-63d2-49b9-84da-ee9d0914d377","Type":"ContainerDied","Data":"271453b76f0e795f8bfd8d00b03bdecc127b7f89283e1bc9ad279904af29aa8f"} Nov 03 00:16:06.977585 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.977533 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:16:06.998978 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:06.998958 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:16:07.065055 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.065022 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc16e901-e277-46c6-96f7-2cf983dd9e03-kube-api-access-ccfrd\" (UniqueName: \"kubernetes.io/projected/bc16e901-e277-46c6-96f7-2cf983dd9e03-kube-api-access-ccfrd\") pod \"bc16e901-e277-46c6-96f7-2cf983dd9e03\" (UID: \"bc16e901-e277-46c6-96f7-2cf983dd9e03\") " Nov 03 00:16:07.065238 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.065092 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e31dcbe-63d2-49b9-84da-ee9d0914d377-kube-api-access-9q2dp\" (UniqueName: \"kubernetes.io/projected/8e31dcbe-63d2-49b9-84da-ee9d0914d377-kube-api-access-9q2dp\") pod \"8e31dcbe-63d2-49b9-84da-ee9d0914d377\" (UID: \"8e31dcbe-63d2-49b9-84da-ee9d0914d377\") " Nov 03 00:16:07.065238 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.065149 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc16e901-e277-46c6-96f7-2cf983dd9e03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc16e901-e277-46c6-96f7-2cf983dd9e03-plugins-config-volume\") pod \"bc16e901-e277-46c6-96f7-2cf983dd9e03\" (UID: \"bc16e901-e277-46c6-96f7-2cf983dd9e03\") " Nov 03 00:16:07.065238 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.065210 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e31dcbe-63d2-49b9-84da-ee9d0914d377-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e31dcbe-63d2-49b9-84da-ee9d0914d377-plugins-config-volume\") pod \"8e31dcbe-63d2-49b9-84da-ee9d0914d377\" (UID: \"8e31dcbe-63d2-49b9-84da-ee9d0914d377\") " Nov 03 00:16:07.065534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.065510 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc16e901-e277-46c6-96f7-2cf983dd9e03-plugins-config-volume" pod "bc16e901-e277-46c6-96f7-2cf983dd9e03" (UID: "bc16e901-e277-46c6-96f7-2cf983dd9e03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:07.065618 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.065598 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e31dcbe-63d2-49b9-84da-ee9d0914d377-plugins-config-volume" pod "8e31dcbe-63d2-49b9-84da-ee9d0914d377" (UID: "8e31dcbe-63d2-49b9-84da-ee9d0914d377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:07.067288 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.067269 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e31dcbe-63d2-49b9-84da-ee9d0914d377-kube-api-access-9q2dp" pod "8e31dcbe-63d2-49b9-84da-ee9d0914d377" (UID: "8e31dcbe-63d2-49b9-84da-ee9d0914d377"). InnerVolumeSpecName "kube-api-access-9q2dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:07.067348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.067332 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc16e901-e277-46c6-96f7-2cf983dd9e03-kube-api-access-ccfrd" pod "bc16e901-e277-46c6-96f7-2cf983dd9e03" (UID: "bc16e901-e277-46c6-96f7-2cf983dd9e03"). InnerVolumeSpecName "kube-api-access-ccfrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:07.166785 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.166752 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc16e901-e277-46c6-96f7-2cf983dd9e03-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:07.166785 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.166792 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e31dcbe-63d2-49b9-84da-ee9d0914d377-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:07.167051 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.166801 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccfrd\" (UniqueName: \"kubernetes.io/projected/bc16e901-e277-46c6-96f7-2cf983dd9e03-kube-api-access-ccfrd\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:07.167051 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.166810 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q2dp\" (UniqueName: \"kubernetes.io/projected/8e31dcbe-63d2-49b9-84da-ee9d0914d377-kube-api-access-9q2dp\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:07.246935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.246859 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t"] Nov 03 00:16:07.247245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247233 4304 cpu_manager.go:414] "containerMap: removing container" podUID="8e31dcbe-63d2-49b9-84da-ee9d0914d377" containerName="payload-pre-processing" Nov 03 00:16:07.247286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247246 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e31dcbe-63d2-49b9-84da-ee9d0914d377" containerName="payload-pre-processing" Nov 03 00:16:07.247286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247256 4304 cpu_manager.go:414] "containerMap: removing container" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" Nov 03 00:16:07.247286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247261 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" Nov 03 00:16:07.247286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247270 4304 cpu_manager.go:414] "containerMap: removing container" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" Nov 03 00:16:07.247286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247275 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" Nov 03 00:16:07.247435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247297 4304 cpu_manager.go:414] "containerMap: removing container" podUID="bc16e901-e277-46c6-96f7-2cf983dd9e03" containerName="payload-processing" Nov 03 00:16:07.247435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247302 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc16e901-e277-46c6-96f7-2cf983dd9e03" containerName="payload-processing" Nov 03 00:16:07.247435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247362 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc16e901-e277-46c6-96f7-2cf983dd9e03" containerName="payload-processing" Nov 03 00:16:07.247435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247370 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e31dcbe-63d2-49b9-84da-ee9d0914d377" containerName="payload-pre-processing" Nov 03 00:16:07.247435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247376 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" Nov 03 00:16:07.247435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.247382 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="135c8d1b-e9c2-4f5a-8b05-ea277767f83b" containerName="istio-proxy" Nov 03 00:16:07.250965 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.250942 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.267251 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.267224 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.267391 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.267253 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.267464 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.267397 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.267464 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.267433 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrsvw\" (UniqueName: \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-kube-api-access-hrsvw\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.267575 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.267465 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.267633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.267573 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.267633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.267614 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.267744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.267643 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.267744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.267670 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.313286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.313201 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t"] Nov 03 00:16:07.369226 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369195 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.369226 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369227 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.369438 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369292 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.369438 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369311 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrsvw\" (UniqueName: \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-kube-api-access-hrsvw\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.369438 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369371 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.369562 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369437 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.369562 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369468 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.369562 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369496 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.369765 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369597 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.369765 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369641 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.369765 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369646 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.370003 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.369985 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.370071 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.370052 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.370494 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.370463 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.371960 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.371878 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.372375 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.372358 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.392376 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.392336 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-531ee4-openshift-default-dockercfg-pcjd8\"" Nov 03 00:16:07.430101 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.430070 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.430237 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.430131 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrsvw\" (UniqueName: \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-kube-api-access-hrsvw\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.564140 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.564112 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:07.929584 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.929546 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" Nov 03 00:16:07.929769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.929547 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp" event={"ID":"bc16e901-e277-46c6-96f7-2cf983dd9e03","Type":"ContainerDied","Data":"86d6fa4ca9d57aa61bf78bcd882606d11980b3ee1c7ba968603ebff252bca06b"} Nov 03 00:16:07.929769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.929673 4304 scope.go:122] "RemoveContainer" containerID="32ac2f4381308c55934091d26fd0657d9afd703547e025190d87c9b3c5247688" Nov 03 00:16:07.931286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.931266 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" event={"ID":"19c95304-1ac6-4047-ad06-09946b5161dd","Type":"ContainerStarted","Data":"7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49"} Nov 03 00:16:07.931400 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.931332 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:07.932639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.932621 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" event={"ID":"8e31dcbe-63d2-49b9-84da-ee9d0914d377","Type":"ContainerDied","Data":"604b5e7deca62362a870ebb1407015686bd2c3696a5d80e69c5a46e1a5b53103"} Nov 03 00:16:07.932712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.932645 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6" Nov 03 00:16:07.939697 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.939676 4304 scope.go:122] "RemoveContainer" containerID="271453b76f0e795f8bfd8d00b03bdecc127b7f89283e1bc9ad279904af29aa8f" Nov 03 00:16:07.990295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.990276 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t"] Nov 03 00:16:07.992739 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:16:07.992712 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a82d52_a209_48e0_bab6_e68c0c36e6e2.slice/crio-fd075607721464a6e58cb5890a1d8bc0aa04e00ccbcb549ec6fafaf74185118a WatchSource:0}: Error finding container fd075607721464a6e58cb5890a1d8bc0aa04e00ccbcb549ec6fafaf74185118a: Status 404 returned error can't find the container with id fd075607721464a6e58cb5890a1d8bc0aa04e00ccbcb549ec6fafaf74185118a Nov 03 00:16:07.995096 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.995065 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:07.995206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.995132 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:07.995206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:07.995163 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:08.093473 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:08.093441 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6"] Nov 03 00:16:08.096859 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:08.096832 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6f27688-6c6db457dc-shzk6"] Nov 03 00:16:08.212622 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:08.212531 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" podStartSLOduration=2.285258214 podStartE2EDuration="3.212516294s" totalImagesPullingTime="927.25808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:08.211007102 +0000 UTC m=+651.494564776" watchObservedRunningTime="2026-11-03 00:16:08.212516294 +0000 UTC m=+651.496073967" Nov 03 00:16:08.322413 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:08.322346 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp"] Nov 03 00:16:08.325603 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:08.325574 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6f27688-7674c56845-lqncp"] Nov 03 00:16:08.828112 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:08.828072 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e31dcbe-63d2-49b9-84da-ee9d0914d377" path="/var/lib/kubelet/pods/8e31dcbe-63d2-49b9-84da-ee9d0914d377/volumes" Nov 03 00:16:08.828616 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:08.828596 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc16e901-e277-46c6-96f7-2cf983dd9e03" path="/var/lib/kubelet/pods/bc16e901-e277-46c6-96f7-2cf983dd9e03/volumes" Nov 03 00:16:08.939696 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:08.939658 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" event={"ID":"e2a82d52-a209-48e0-bab6-e68c0c36e6e2","Type":"ContainerStarted","Data":"8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a"} Nov 03 00:16:08.939904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:08.939715 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" event={"ID":"e2a82d52-a209-48e0-bab6-e68c0c36e6e2","Type":"ContainerStarted","Data":"fd075607721464a6e58cb5890a1d8bc0aa04e00ccbcb549ec6fafaf74185118a"} Nov 03 00:16:09.069289 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:09.069238 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" podStartSLOduration=2.06922393 podStartE2EDuration="2.06922393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:09.068607838 +0000 UTC m=+652.352165512" watchObservedRunningTime="2026-11-03 00:16:09.06922393 +0000 UTC m=+652.352781604" Nov 03 00:16:09.564520 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:09.564488 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:09.569733 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:09.569707 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:09.946478 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:09.946444 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:09.947315 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:09.947297 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:12.896682 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:12.896658 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 00:16:12.897158 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:12.896731 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 00:16:13.947584 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:13.947556 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 00:16:15.851096 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:15.851062 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z"] Nov 03 00:16:15.855566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:15.855546 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:15.908223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:15.908195 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ebeeb9-serving-cert\"" Nov 03 00:16:15.947911 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:15.947879 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:15.948049 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:15.947997 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:15.948094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:15.948059 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:15.948094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:15.948086 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwrd\" (UniqueName: \"kubernetes.io/projected/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-kube-api-access-qtwrd\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:15.959414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:15.959384 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z"] Nov 03 00:16:16.049671 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:16.049644 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwrd\" (UniqueName: \"kubernetes.io/projected/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-kube-api-access-qtwrd\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:16.049886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:16.049741 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:16.049886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:16.049826 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:16.049886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:16.049876 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:16.050078 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:16:16.049952 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-serving-cert: secret "maas-api-e2e-worker-w0-ebeeb9-serving-cert" not found Nov 03 00:16:16.050078 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:16:16.050037 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-maas-api-tls podName:4a4c84b0-2ca5-4173-a51a-83116c37bc8e nodeName:}" failed. No retries permitted until 2026-11-03 00:16:16.550015723 +0000 UTC m=+659.833573391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-maas-api-tls") pod "maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" (UID: "4a4c84b0-2ca5-4173-a51a-83116c37bc8e") : secret "maas-api-e2e-worker-w0-ebeeb9-serving-cert" not found Nov 03 00:16:16.050525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:16.050505 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:16.052230 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:16.052213 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:16.106704 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:16.106643 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwrd\" (UniqueName: \"kubernetes.io/projected/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-kube-api-access-qtwrd\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:16.554407 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:16.554368 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:16.557152 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:16.557116 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:16.767989 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:16.767955 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:17.140857 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:17.140821 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z"] Nov 03 00:16:17.990366 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:17.990332 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" event={"ID":"4a4c84b0-2ca5-4173-a51a-83116c37bc8e","Type":"ContainerStarted","Data":"2dda8ac69b2003d3f5b10966d186ad7dddb856ac6334fde1242eb057412f451e"} Nov 03 00:16:18.996680 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:18.996646 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" event={"ID":"4a4c84b0-2ca5-4173-a51a-83116c37bc8e","Type":"ContainerStarted","Data":"d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe"} Nov 03 00:16:18.997129 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:18.996809 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:19.112895 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:19.112846 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" podStartSLOduration=3.29696446 podStartE2EDuration="4.112831946s" totalImagesPullingTime="815.867486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:19.110721806 +0000 UTC m=+662.394279493" watchObservedRunningTime="2026-11-03 00:16:19.112831946 +0000 UTC m=+662.396389620" Nov 03 00:16:24.985020 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:24.984985 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l"] Nov 03 00:16:24.990148 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:24.990124 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.007404 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.007378 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 00:16:25.042085 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.042031 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d7bfc2-openshift-default-dockercfg-g56sm\"" Nov 03 00:16:25.064462 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.064430 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.064633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.064479 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.064633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.064513 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.064727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.064684 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.064820 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.064748 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.064892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.064855 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58twv\" (UniqueName: \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-kube-api-access-58twv\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.064949 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.064905 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.064949 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.064940 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.065041 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.065016 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf51c557-8e65-4a4d-8746-35df22b465e6-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.094096 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.094064 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l"] Nov 03 00:16:25.165719 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.165690 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf51c557-8e65-4a4d-8746-35df22b465e6-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.165719 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.165731 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.165757 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.165829 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.166019 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.166098 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166217 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.166133 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58twv\" (UniqueName: \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-kube-api-access-58twv\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166217 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.166168 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166217 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.166172 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.166238 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166497 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.166471 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166817 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.166769 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166919 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.166802 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.166969 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.166864 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf51c557-8e65-4a4d-8746-35df22b465e6-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.168598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.168575 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.168807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.168764 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.249670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.248747 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.249884 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.249799 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58twv\" (UniqueName: \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-kube-api-access-58twv\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.305247 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.305201 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:25.764391 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.764358 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l"] Nov 03 00:16:25.767060 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:16:25.767033 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf51c557_8e65_4a4d_8746_35df22b465e6.slice/crio-38fa61bab29dfb8a6cd77e886ee3526af1dc822c174783c69217706fe0107182 WatchSource:0}: Error finding container 38fa61bab29dfb8a6cd77e886ee3526af1dc822c174783c69217706fe0107182: Status 404 returned error can't find the container with id 38fa61bab29dfb8a6cd77e886ee3526af1dc822c174783c69217706fe0107182 Nov 03 00:16:25.769276 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.769242 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:25.769365 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.769306 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:25.769365 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:25.769337 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:26.056703 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:26.056614 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" event={"ID":"bf51c557-8e65-4a4d-8746-35df22b465e6","Type":"ContainerStarted","Data":"3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856"} Nov 03 00:16:26.056703 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:26.056680 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" event={"ID":"bf51c557-8e65-4a4d-8746-35df22b465e6","Type":"ContainerStarted","Data":"38fa61bab29dfb8a6cd77e886ee3526af1dc822c174783c69217706fe0107182"} Nov 03 00:16:26.177087 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:26.177038 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" podStartSLOduration=2.177022519 podStartE2EDuration="2.177022519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:26.174717984 +0000 UTC m=+669.458275672" watchObservedRunningTime="2026-11-03 00:16:26.177022519 +0000 UTC m=+669.460580374" Nov 03 00:16:26.305640 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:26.305599 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:26.311604 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:26.311547 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:27.061862 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:27.061830 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:27.062823 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:27.062801 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:16:28.068279 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:28.068246 4304 generic.go:356] "Generic (PLEG): container finished" podID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerID="fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0" exitCode=0 Nov 03 00:16:28.068728 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:28.068323 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" event={"ID":"b31f9e36-ddf8-41a7-abcf-10d14702cb35","Type":"ContainerDied","Data":"fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0"} Nov 03 00:16:28.068842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:28.068767 4304 scope.go:122] "RemoveContainer" containerID="fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0" Nov 03 00:16:28.069931 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:28.069908 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:28.070000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:28.069979 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:28.070040 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:28.070013 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:28.693945 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:16:28.693906 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:16:29.084994 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:29.084911 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" event={"ID":"b31f9e36-ddf8-41a7-abcf-10d14702cb35","Type":"ContainerStarted","Data":"f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61"} Nov 03 00:16:29.883018 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:29.882983 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:16:29.883186 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:29.883165 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 00:16:29.883230 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:29.883201 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 00:16:30.883432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:30.883400 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 00:16:30.883844 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:30.883453 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 00:16:31.883663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:31.883619 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 00:16:31.884201 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:31.883681 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 00:16:32.163531 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.162039 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57"] Nov 03 00:16:32.165567 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.165535 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.217632 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.217601 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-531ee4-serving-cert\"" Nov 03 00:16:32.260637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.260594 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-maas-api-tls\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.260832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.260658 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p9dv\" (UniqueName: \"kubernetes.io/projected/b83b4462-07b9-4c5a-8acb-8e7cb777d794-kube-api-access-4p9dv\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.260832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.260758 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83b4462-07b9-4c5a-8acb-8e7cb777d794-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.260935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.260906 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-metrics-certs\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.270537 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.270505 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57"] Nov 03 00:16:32.362470 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.362428 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-maas-api-tls\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.362470 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.362472 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p9dv\" (UniqueName: \"kubernetes.io/projected/b83b4462-07b9-4c5a-8acb-8e7cb777d794-kube-api-access-4p9dv\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.362719 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.362499 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83b4462-07b9-4c5a-8acb-8e7cb777d794-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.362719 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:16:32.362590 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-serving-cert: secret "maas-api-e2e-worker-w2-531ee4-serving-cert" not found Nov 03 00:16:32.362719 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.362601 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-metrics-certs\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.362719 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:16:32.362658 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-maas-api-tls podName:b83b4462-07b9-4c5a-8acb-8e7cb777d794 nodeName:}" failed. No retries permitted until 2026-11-03 00:16:32.862638538 +0000 UTC m=+676.146196204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-maas-api-tls") pod "maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" (UID: "b83b4462-07b9-4c5a-8acb-8e7cb777d794") : secret "maas-api-e2e-worker-w2-531ee4-serving-cert" not found Nov 03 00:16:32.363293 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.363255 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83b4462-07b9-4c5a-8acb-8e7cb777d794-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.365217 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.365195 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-metrics-certs\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.458289 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.450602 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p9dv\" (UniqueName: \"kubernetes.io/projected/b83b4462-07b9-4c5a-8acb-8e7cb777d794-kube-api-access-4p9dv\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.868093 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.868003 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-maas-api-tls\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.870523 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.870499 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-maas-api-tls\") pod \"maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:32.883242 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.883209 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 00:16:32.883372 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:32.883255 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 00:16:33.080036 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:33.079997 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:33.481434 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:33.481406 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57"] Nov 03 00:16:33.482737 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:16:33.482711 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83b4462_07b9_4c5a_8acb_8e7cb777d794.slice/crio-0a4c2dbace386376065085d225bd2f8a8e945ea16eca267ca215e3aa4ec25888 WatchSource:0}: Error finding container 0a4c2dbace386376065085d225bd2f8a8e945ea16eca267ca215e3aa4ec25888: Status 404 returned error can't find the container with id 0a4c2dbace386376065085d225bd2f8a8e945ea16eca267ca215e3aa4ec25888 Nov 03 00:16:33.883250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:33.883162 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 00:16:33.883250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:33.883227 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 00:16:34.116531 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:34.116497 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" event={"ID":"b83b4462-07b9-4c5a-8acb-8e7cb777d794","Type":"ContainerStarted","Data":"0a4c2dbace386376065085d225bd2f8a8e945ea16eca267ca215e3aa4ec25888"} Nov 03 00:16:34.887828 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:34.887766 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:16:34.888168 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:34.888140 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:16:34.888755 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:34.888738 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 00:16:35.124980 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:35.124947 4304 generic.go:356] "Generic (PLEG): container finished" podID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerID="27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9" exitCode=0 Nov 03 00:16:35.125473 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:35.125029 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" event={"ID":"6dabb012-d842-4c6b-8937-de5b400dbc46","Type":"ContainerDied","Data":"27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9"} Nov 03 00:16:35.125535 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:35.125484 4304 scope.go:122] "RemoveContainer" containerID="27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9" Nov 03 00:16:35.126833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:35.126808 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" event={"ID":"b83b4462-07b9-4c5a-8acb-8e7cb777d794","Type":"ContainerStarted","Data":"1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20"} Nov 03 00:16:35.126991 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:35.126974 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:35.127108 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:35.127085 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:35.127169 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:35.127149 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:35.127236 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:35.127181 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:35.353278 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:35.353211 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" podStartSLOduration=2.575950651 podStartE2EDuration="3.353193212s" totalImagesPullingTime="777.242561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:35.352643446 +0000 UTC m=+678.636201120" watchObservedRunningTime="2026-11-03 00:16:35.353193212 +0000 UTC m=+678.636750889" Nov 03 00:16:36.151989 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:36.151955 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" event={"ID":"6dabb012-d842-4c6b-8937-de5b400dbc46","Type":"ContainerStarted","Data":"0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab"} Nov 03 00:16:36.495260 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:36.495183 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:16:36.496241 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:36.496219 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 00:16:36.496305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:36.496280 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 00:16:37.495944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:37.495902 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 00:16:37.496338 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:37.495971 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 00:16:37.820114 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:37.820039 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:16:38.495577 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:38.495537 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 00:16:38.495764 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:38.495594 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 00:16:39.496063 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:39.496028 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 00:16:39.496451 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:39.496083 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 00:16:40.495616 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:40.495582 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 00:16:40.495805 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:40.495633 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 00:16:41.159392 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:41.159363 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 00:16:41.500519 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:41.500445 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:16:41.501435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:41.501412 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:16:50.238363 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:50.238329 4304 generic.go:356] "Generic (PLEG): container finished" podID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerID="8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a" exitCode=0 Nov 03 00:16:50.238859 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:50.238401 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" event={"ID":"e2a82d52-a209-48e0-bab6-e68c0c36e6e2","Type":"ContainerDied","Data":"8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a"} Nov 03 00:16:50.238859 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:50.238790 4304 scope.go:122] "RemoveContainer" containerID="8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a" Nov 03 00:16:50.239983 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:50.239958 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:50.240045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:50.240022 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:50.240080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:50.240053 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:51.250659 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:51.250620 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" event={"ID":"e2a82d52-a209-48e0-bab6-e68c0c36e6e2","Type":"ContainerStarted","Data":"d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a"} Nov 03 00:16:51.565335 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:51.565247 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:51.566640 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:51.566605 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:16:51.566821 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:51.566671 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 00:16:52.564734 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:52.564699 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:16:52.565160 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:52.564751 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 00:16:53.565121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:53.565086 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:16:53.565514 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:53.565146 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 00:16:54.564654 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:54.564621 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:16:54.564848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:54.564671 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 00:16:54.947133 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:54.947060 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:55.565533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:55.565492 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:16:55.565728 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:55.565550 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 00:16:56.569083 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:56.569048 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:56.570271 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:16:56.570248 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 00:16:58.654434 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:16:58.654392 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:17:12.063252 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:12.063154 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:17:12.063252 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:12.063213 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 00:17:12.390033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:12.390003 4304 generic.go:356] "Generic (PLEG): container finished" podID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerID="3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856" exitCode=0 Nov 03 00:17:12.390189 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:12.390087 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" event={"ID":"bf51c557-8e65-4a4d-8746-35df22b465e6","Type":"ContainerDied","Data":"3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856"} Nov 03 00:17:12.390440 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:12.390423 4304 scope.go:122] "RemoveContainer" containerID="3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856" Nov 03 00:17:12.391650 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:12.391621 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:17:12.391756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:12.391707 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:17:12.391860 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:12.391757 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:17:13.397960 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:13.397929 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" event={"ID":"bf51c557-8e65-4a4d-8746-35df22b465e6","Type":"ContainerStarted","Data":"7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b"} Nov 03 00:17:14.305698 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:14.305665 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:17:14.306024 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:14.306002 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:17:14.306087 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:14.306068 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 00:17:15.306620 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:15.306583 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:17:15.307043 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:15.306636 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 00:17:16.305803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:16.305757 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:17:16.305980 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:16.305825 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 00:17:17.306056 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:17.306018 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:17:17.306477 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:17.306075 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 00:17:18.306250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:18.306217 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:17:18.306642 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:18.306281 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 00:17:19.310097 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:19.310061 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:17:19.310586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:19.310488 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:17:19.311536 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:19.311509 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 00:17:28.654216 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:17:28.654182 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:17:58.655746 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:17:58.655705 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:17:59.124075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.123999 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj"] Nov 03 00:17:59.128030 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.128009 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.224566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.224527 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-dshm\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.224756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.224592 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfk7w\" (UniqueName: \"kubernetes.io/projected/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-kube-api-access-kfk7w\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.224756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.224703 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-model-cache\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.224756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.224737 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tls-certs\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.224967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.224762 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tmp-dir\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.224967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.224837 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-home\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.231906 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.231883 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"openshift-service-ca.crt\"" Nov 03 00:17:59.231906 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.231898 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"kube-root-ca.crt\"" Nov 03 00:17:59.232085 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.231960 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"default-dockercfg-mpmcw\"" Nov 03 00:17:59.232085 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.231996 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"simulator-w3-d7bfc2-kserve-self-signed-certs\"" Nov 03 00:17:59.234241 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.234217 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj"] Nov 03 00:17:59.325501 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.325423 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfk7w\" (UniqueName: \"kubernetes.io/projected/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-kube-api-access-kfk7w\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.325501 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.325463 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-model-cache\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.325501 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.325487 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tls-certs\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.325810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.325511 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tmp-dir\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.325810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.325698 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-home\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.325920 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.325877 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-dshm\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.325920 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.325896 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-model-cache\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.326232 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.325914 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tmp-dir\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.326232 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.326059 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-home\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.328131 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.328105 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-dshm\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.328311 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.328293 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tls-certs\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.381908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.381832 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfk7w\" (UniqueName: \"kubernetes.io/projected/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-kube-api-access-kfk7w\") pod \"simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.438838 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.438799 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:17:59.810372 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:17:59.810349 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj"] Nov 03 00:17:59.811449 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:17:59.811427 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod082c2451_3bb7_4b3a_82c0_f8ea634adcdb.slice/crio-9de8f95230a1d8c6b9bab0d0a0e2dd5b3c99ac30adf5aa4c801b9b9284ed70fd WatchSource:0}: Error finding container 9de8f95230a1d8c6b9bab0d0a0e2dd5b3c99ac30adf5aa4c801b9b9284ed70fd: Status 404 returned error can't find the container with id 9de8f95230a1d8c6b9bab0d0a0e2dd5b3c99ac30adf5aa4c801b9b9284ed70fd Nov 03 00:18:00.649125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:00.649090 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" event={"ID":"082c2451-3bb7-4b3a-82c0-f8ea634adcdb","Type":"ContainerStarted","Data":"7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766"} Nov 03 00:18:00.649125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:00.649129 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" event={"ID":"082c2451-3bb7-4b3a-82c0-f8ea634adcdb","Type":"ContainerStarted","Data":"9de8f95230a1d8c6b9bab0d0a0e2dd5b3c99ac30adf5aa4c801b9b9284ed70fd"} Nov 03 00:18:00.763704 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:00.763655 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" podStartSLOduration=1.763640647 podStartE2EDuration="1.763640647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:00.761874729 +0000 UTC m=+764.045432403" watchObservedRunningTime="2026-11-03 00:18:00.763640647 +0000 UTC m=+764.047198321" Nov 03 00:18:01.439446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:01.439408 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:18:09.439099 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:09.439064 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:18:09.451313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:09.451283 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:18:09.706591 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:09.706506 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:18:28.654201 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:18:28.654164 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:18:40.760177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.760144 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq"] Nov 03 00:18:40.763752 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.763736 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.837758 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.837729 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d57814-f91a-423c-ab38-5cb55e4d77a0-tls-certs\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.837930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.837769 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzxmc\" (UniqueName: \"kubernetes.io/projected/26d57814-f91a-423c-ab38-5cb55e4d77a0-kube-api-access-qzxmc\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.837930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.837877 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-tmp-dir\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.837930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.837915 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-model-cache\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.838060 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.837996 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-home\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.838060 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.838038 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-dshm\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.876025 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.875978 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"kube-root-ca.crt\"" Nov 03 00:18:40.876025 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.875985 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"premium-w2-531ee4-kserve-self-signed-certs\"" Nov 03 00:18:40.876228 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.876039 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"openshift-service-ca.crt\"" Nov 03 00:18:40.876228 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.876056 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"default-dockercfg-br2w9\"" Nov 03 00:18:40.879039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.879015 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq"] Nov 03 00:18:40.939081 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.939047 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-tmp-dir\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.939276 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.939099 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-model-cache\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.939276 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.939162 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-home\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.939276 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.939208 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-dshm\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.939447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.939287 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d57814-f91a-423c-ab38-5cb55e4d77a0-tls-certs\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.939447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.939327 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxmc\" (UniqueName: \"kubernetes.io/projected/26d57814-f91a-423c-ab38-5cb55e4d77a0-kube-api-access-qzxmc\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.939543 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.939489 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-model-cache\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.939543 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.939521 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-tmp-dir\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.939543 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.939534 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-home\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.941529 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.941504 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-dshm\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:40.941909 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:40.941888 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d57814-f91a-423c-ab38-5cb55e4d77a0-tls-certs\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:41.014893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:41.014834 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxmc\" (UniqueName: \"kubernetes.io/projected/26d57814-f91a-423c-ab38-5cb55e4d77a0-kube-api-access-qzxmc\") pod \"premium-w2-531ee4-kserve-7d445c97fc-rvvfq\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:41.075459 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:41.075430 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:41.449229 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:41.449205 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq"] Nov 03 00:18:41.450968 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:18:41.450943 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26d57814_f91a_423c_ab38_5cb55e4d77a0.slice/crio-520771a47aab8fec22235c67b98940c0cec558152c96d73bf07f1eeb62e49b91 WatchSource:0}: Error finding container 520771a47aab8fec22235c67b98940c0cec558152c96d73bf07f1eeb62e49b91: Status 404 returned error can't find the container with id 520771a47aab8fec22235c67b98940c0cec558152c96d73bf07f1eeb62e49b91 Nov 03 00:18:41.858628 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:41.858538 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" event={"ID":"26d57814-f91a-423c-ab38-5cb55e4d77a0","Type":"ContainerStarted","Data":"1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad"} Nov 03 00:18:41.858628 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:41.858577 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" event={"ID":"26d57814-f91a-423c-ab38-5cb55e4d77a0","Type":"ContainerStarted","Data":"520771a47aab8fec22235c67b98940c0cec558152c96d73bf07f1eeb62e49b91"} Nov 03 00:18:41.973528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:41.973480 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" podStartSLOduration=1.9734672070000001 podStartE2EDuration="1.973467207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:41.971479159 +0000 UTC m=+805.255036832" watchObservedRunningTime="2026-11-03 00:18:41.973467207 +0000 UTC m=+805.257024880" Nov 03 00:18:42.076569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:42.076537 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:51.075939 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:51.075907 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:51.088484 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:51.088463 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:51.919721 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:18:51.919692 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 00:18:58.655941 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:18:58.655901 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:19:28.654958 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:19:28.654915 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:19:58.654312 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:19:58.654235 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:20:22.433749 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.433714 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf"] Nov 03 00:20:22.437433 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.437417 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.459568 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.459532 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.459568 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.459564 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.459769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.459677 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.459769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.459737 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.459899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.459846 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.459965 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.459934 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gqmz\" (UniqueName: \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-kube-api-access-4gqmz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.460085 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.459992 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.460085 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.460040 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.460257 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.460147 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.541794 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.541744 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf"] Nov 03 00:20:22.561363 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.561329 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.561565 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.561417 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.561565 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.561447 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.561565 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.561524 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.561754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.561576 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.561754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.561606 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.561754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.561647 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gqmz\" (UniqueName: \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-kube-api-access-4gqmz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.561754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.561688 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.561754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.561739 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.562045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.561929 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.562131 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.562108 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.562375 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.562350 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.562624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.562601 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.563044 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.563014 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.564363 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.564339 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.564468 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.564445 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.618398 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.618361 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.618547 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.618498 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gqmz\" (UniqueName: \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-kube-api-access-4gqmz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:22.750403 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:22.750323 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:23.173842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:23.173815 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf"] Nov 03 00:20:23.174988 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:20:23.174963 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd0d831_a1b5_4ff0_ab36_726b1c1762ae.slice/crio-ddc8c71cb1fa6b864c16de5fcdc4d56d414aa52e11c6b3e0e786d1c9fb7c45cf WatchSource:0}: Error finding container ddc8c71cb1fa6b864c16de5fcdc4d56d414aa52e11c6b3e0e786d1c9fb7c45cf: Status 404 returned error can't find the container with id ddc8c71cb1fa6b864c16de5fcdc4d56d414aa52e11c6b3e0e786d1c9fb7c45cf Nov 03 00:20:23.177136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:23.177106 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:23.177256 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:23.177186 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:23.177256 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:23.177231 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:23.407349 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:23.407314 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" event={"ID":"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae","Type":"ContainerStarted","Data":"51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84"} Nov 03 00:20:23.407349 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:23.407369 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" event={"ID":"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae","Type":"ContainerStarted","Data":"ddc8c71cb1fa6b864c16de5fcdc4d56d414aa52e11c6b3e0e786d1c9fb7c45cf"} Nov 03 00:20:23.522625 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:23.522506 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" podStartSLOduration=1.5224882499999999 podStartE2EDuration="1.52248825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:23.522307101 +0000 UTC m=+906.805864787" watchObservedRunningTime="2026-11-03 00:20:23.52248825 +0000 UTC m=+906.806045925" Nov 03 00:20:23.751939 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:23.751900 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:24.752713 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:24.752677 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:20:24.753152 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:24.752742 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 00:20:25.752225 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:25.752189 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:20:25.752418 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:25.752255 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 00:20:26.751165 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:26.751133 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:20:26.751542 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:26.751188 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 00:20:27.751692 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:27.751656 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:20:27.752089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:27.751713 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 00:20:28.670421 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:20:28.670376 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:20:28.751616 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:28.751584 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:20:28.751804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:28.751640 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 00:20:29.751243 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:29.751213 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:20:29.751450 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:29.751261 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 00:20:31.566553 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:31.566524 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:31.567288 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:31.567009 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:31.567988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:31.567970 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:20:37.554352 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.554311 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94"] Nov 03 00:20:37.557900 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.557884 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.604085 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.604058 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5"] Nov 03 00:20:37.607754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.607739 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.649168 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.649144 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.649317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.649173 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.649317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.649206 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.649440 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.649392 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.649440 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.649430 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.649515 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.649485 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.649558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.649535 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gz2\" (UniqueName: \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-kube-api-access-c8gz2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.649601 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.649561 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.649601 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.649594 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.660512 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.660489 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94"] Nov 03 00:20:37.712471 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.712441 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5"] Nov 03 00:20:37.750533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.750494 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.750690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.750545 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.750690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.750622 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.750690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.750640 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.750690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.750666 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.750690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.750685 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.750890 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.750744 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.750890 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.750811 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gz2\" (UniqueName: \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-kube-api-access-c8gz2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.750890 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.750845 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.750890 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.750870 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.751079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.750897 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.751079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751049 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.751183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751093 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtgnc\" (UniqueName: \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-kube-api-access-gtgnc\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.751245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751186 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.751245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751217 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.751359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751263 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.751359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751338 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.751455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751382 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.751658 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751638 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.751700 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751685 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.751734 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751703 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.751767 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751730 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.751767 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.751733 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.753212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.753196 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.753431 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.753413 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.807820 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.807736 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.808088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.808063 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gz2\" (UniqueName: \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-kube-api-access-c8gz2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4zf94\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.862352 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.862414 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.862552 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.862610 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.862670 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.862721 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.862756 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgnc\" (UniqueName: \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-kube-api-access-gtgnc\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.862840 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.862880 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.863325 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.863562 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.863810 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.864799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.864564 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.865598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.865155 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.868267 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.868228 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.881860 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.876344 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:37.881860 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.877088 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.920479 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.920448 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtgnc\" (UniqueName: \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-kube-api-access-gtgnc\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:37.920919 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:37.920899 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r66p5\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:38.219971 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.219936 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:38.303493 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.303459 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94"] Nov 03 00:20:38.304533 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:20:38.304510 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054e4951_35e0_44e9_a75d_a6d4cb457b1d.slice/crio-834f56a76fe11573a7eafb7b6fe24a84a0488ba813c252fdb07b5e54056a664b WatchSource:0}: Error finding container 834f56a76fe11573a7eafb7b6fe24a84a0488ba813c252fdb07b5e54056a664b: Status 404 returned error can't find the container with id 834f56a76fe11573a7eafb7b6fe24a84a0488ba813c252fdb07b5e54056a664b Nov 03 00:20:38.307625 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.307595 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:38.307715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.307666 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:38.307715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.307695 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:38.520092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.520009 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" event={"ID":"054e4951-35e0-44e9-a75d-a6d4cb457b1d","Type":"ContainerStarted","Data":"73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb"} Nov 03 00:20:38.520092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.520065 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" event={"ID":"054e4951-35e0-44e9-a75d-a6d4cb457b1d","Type":"ContainerStarted","Data":"834f56a76fe11573a7eafb7b6fe24a84a0488ba813c252fdb07b5e54056a664b"} Nov 03 00:20:38.594590 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.594566 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5"] Nov 03 00:20:38.599753 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:20:38.599717 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf456d8be_bb77_48f5_83fa_f684bcbc6cb9.slice/crio-d127feefe47b3ff2e7deedb18a9ec6eee0f3c47167df7808ceb659574f5bb3c1 WatchSource:0}: Error finding container d127feefe47b3ff2e7deedb18a9ec6eee0f3c47167df7808ceb659574f5bb3c1: Status 404 returned error can't find the container with id d127feefe47b3ff2e7deedb18a9ec6eee0f3c47167df7808ceb659574f5bb3c1 Nov 03 00:20:38.601628 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.601598 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:38.601718 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.601679 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:38.601718 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.601709 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:38.637120 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.637073 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podStartSLOduration=1.6370577389999998 podStartE2EDuration="1.637057739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:38.635214069 +0000 UTC m=+921.918771757" watchObservedRunningTime="2026-11-03 00:20:38.637057739 +0000 UTC m=+921.920615413" Nov 03 00:20:38.877085 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:38.876995 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:39.559802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:39.558797 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" event={"ID":"f456d8be-bb77-48f5-83fa-f684bcbc6cb9","Type":"ContainerStarted","Data":"b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff"} Nov 03 00:20:39.559802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:39.558857 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" event={"ID":"f456d8be-bb77-48f5-83fa-f684bcbc6cb9","Type":"ContainerStarted","Data":"d127feefe47b3ff2e7deedb18a9ec6eee0f3c47167df7808ceb659574f5bb3c1"} Nov 03 00:20:39.675915 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:39.675860 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" podStartSLOduration=2.6758444040000002 podStartE2EDuration="2.675844404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:39.673279875 +0000 UTC m=+922.956837549" watchObservedRunningTime="2026-11-03 00:20:39.675844404 +0000 UTC m=+922.959402128" Nov 03 00:20:39.879143 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:39.879065 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:20:39.879143 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:39.879118 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 00:20:40.222059 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:40.220615 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:40.227233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:40.226893 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:20:40.227233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:40.226955 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 00:20:40.881937 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:40.881878 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:20:40.882398 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:40.881969 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 00:20:41.220679 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:41.220647 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:20:41.220909 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:41.220711 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 00:20:41.877591 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:41.877545 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:20:41.877800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:41.877625 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 00:20:42.220565 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:42.220530 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:20:42.221038 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:42.220585 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 00:20:42.878249 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:42.878214 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:20:42.878520 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:42.878491 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 00:20:43.226205 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:43.226168 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:20:43.226851 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:43.226817 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 00:20:43.877252 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:43.877212 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:20:43.877431 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:43.877266 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 00:20:44.230075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:44.230037 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:20:44.230709 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:44.230667 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 00:20:44.877721 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:44.877684 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:20:44.877937 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:44.877751 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 00:20:45.226802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:45.224193 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:20:45.226802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:45.224254 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 00:20:45.880566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:45.880124 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:20:45.880566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:45.880190 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 00:20:46.220847 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:46.220811 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:20:46.221119 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:46.221091 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 00:20:47.879522 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:47.879473 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:47.880055 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:47.879537 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:48.220725 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:48.220694 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:48.220942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:48.220761 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:48.920947 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:48.920899 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:48.921550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:48.921512 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:49.262083 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:49.261953 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:49.262253 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:49.262071 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:49.828164 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:49.828126 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:49.828663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:49.828643 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:49.917990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:49.917964 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:20:50.225802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:50.225746 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:50.226316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:50.226290 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:50.227140 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:50.227121 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:20:58.654652 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:20:58.654615 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:20:59.199023 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.198991 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs"] Nov 03 00:20:59.203838 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.203821 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.255967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.255899 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 00:20:59.255967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.255905 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 00:20:59.255967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.255911 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 00:20:59.255967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.255920 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0fb7528a8dea\"" Nov 03 00:20:59.255967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.255955 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pnghk\"" Nov 03 00:20:59.260483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.260462 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-provider-credentials\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.260557 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.260500 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvjk4\" (UniqueName: \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-kube-api-access-qvjk4\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.260688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.260671 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-plugins-config-volume\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.260752 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.260728 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-routing-overlay-volume\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.305511 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.305485 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 00:20:59.310133 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.310104 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs"] Nov 03 00:20:59.340654 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.340630 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc"] Nov 03 00:20:59.344157 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.344142 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.361815 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.361790 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-routing-overlay-volume\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.361904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.361843 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-provider-credentials\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.361904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.361889 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvjk4\" (UniqueName: \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-kube-api-access-qvjk4\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.362055 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.362038 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-plugins-config-volume\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.362583 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.362501 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-routing-overlay-volume\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.362769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.362689 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-plugins-config-volume\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.364357 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.364337 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-provider-credentials\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.399040 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.399007 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"distinct-w4-048135-kserve-self-signed-certs\"" Nov 03 00:20:59.399243 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.399176 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"openshift-service-ca.crt\"" Nov 03 00:20:59.415360 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.415331 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"default-dockercfg-xsf2g\"" Nov 03 00:20:59.415492 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.415374 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"kube-root-ca.crt\"" Nov 03 00:20:59.418426 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.418402 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc"] Nov 03 00:20:59.419372 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.419349 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvjk4\" (UniqueName: \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-kube-api-access-qvjk4\") pod \"payload-processing-external-model-698cf49b84-8qlgs\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.463655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.463594 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhsr\" (UniqueName: \"kubernetes.io/projected/aa4f830f-1dc3-4231-ab46-49e522d2e794-kube-api-access-qmhsr\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.463754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.463659 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-dshm\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.463754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.463724 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f830f-1dc3-4231-ab46-49e522d2e794-tls-certs\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.463854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.463816 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-tmp-dir\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.463896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.463882 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-home\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.463932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.463911 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-model-cache\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.517360 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.517326 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:20:59.565254 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.565219 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f830f-1dc3-4231-ab46-49e522d2e794-tls-certs\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.565254 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.565267 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-tmp-dir\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.565527 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.565300 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-home\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.565527 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.565323 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-model-cache\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.565527 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.565402 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhsr\" (UniqueName: \"kubernetes.io/projected/aa4f830f-1dc3-4231-ab46-49e522d2e794-kube-api-access-qmhsr\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.565527 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.565444 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-dshm\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.565953 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.565927 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-home\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.566034 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.565978 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-tmp-dir\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.566199 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.566177 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-model-cache\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.567687 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.567659 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-dshm\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.567837 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.567818 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f830f-1dc3-4231-ab46-49e522d2e794-tls-certs\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.623315 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.623287 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhsr\" (UniqueName: \"kubernetes.io/projected/aa4f830f-1dc3-4231-ab46-49e522d2e794-kube-api-access-qmhsr\") pod \"distinct-w4-048135-kserve-f665cd4b-kt7zc\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.655899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.655863 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:20:59.900049 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:20:59.900028 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs"] Nov 03 00:20:59.901434 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:20:59.901407 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05704dba_5887_4069_a07f_6e1184ea6ee3.slice/crio-db47b63c2d833dd0214424e7389e6d98fd97a2e40f94e3afe2eac5abe8d2f56b WatchSource:0}: Error finding container db47b63c2d833dd0214424e7389e6d98fd97a2e40f94e3afe2eac5abe8d2f56b: Status 404 returned error can't find the container with id db47b63c2d833dd0214424e7389e6d98fd97a2e40f94e3afe2eac5abe8d2f56b Nov 03 00:21:00.032308 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:00.032283 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc"] Nov 03 00:21:00.033922 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:21:00.033897 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa4f830f_1dc3_4231_ab46_49e522d2e794.slice/crio-a38646422211c7a9069f3eaffbb46b0501faff1af5b50582b05297f3d8a4132c WatchSource:0}: Error finding container a38646422211c7a9069f3eaffbb46b0501faff1af5b50582b05297f3d8a4132c: Status 404 returned error can't find the container with id a38646422211c7a9069f3eaffbb46b0501faff1af5b50582b05297f3d8a4132c Nov 03 00:21:00.700613 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:00.700493 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" event={"ID":"aa4f830f-1dc3-4231-ab46-49e522d2e794","Type":"ContainerStarted","Data":"cdbfae36adfeb86cebe7caebb2c501318f97ca3a62ddf6e9201e6bccea51acb8"} Nov 03 00:21:00.700613 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:00.700552 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" event={"ID":"aa4f830f-1dc3-4231-ab46-49e522d2e794","Type":"ContainerStarted","Data":"a38646422211c7a9069f3eaffbb46b0501faff1af5b50582b05297f3d8a4132c"} Nov 03 00:21:00.702730 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:00.702702 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" event={"ID":"05704dba-5887-4069-a07f-6e1184ea6ee3","Type":"ContainerStarted","Data":"c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15"} Nov 03 00:21:00.702873 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:00.702790 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" event={"ID":"05704dba-5887-4069-a07f-6e1184ea6ee3","Type":"ContainerStarted","Data":"db47b63c2d833dd0214424e7389e6d98fd97a2e40f94e3afe2eac5abe8d2f56b"} Nov 03 00:21:00.702873 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:00.702821 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:21:00.815705 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:00.815646 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" podStartSLOduration=1.815633791 podStartE2EDuration="1.815633791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:00.812920775 +0000 UTC m=+944.096478450" watchObservedRunningTime="2026-11-03 00:21:00.815633791 +0000 UTC m=+944.099191464" Nov 03 00:21:00.929683 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:00.929619 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" podStartSLOduration=1.929598398 podStartE2EDuration="1.929598398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:00.926765916 +0000 UTC m=+944.210323590" watchObservedRunningTime="2026-11-03 00:21:00.929598398 +0000 UTC m=+944.213156089" Nov 03 00:21:01.656039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:01.656000 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:21:09.656354 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:09.656316 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:21:09.670258 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:09.670230 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:21:09.786986 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:09.786958 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:21:11.713351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:11.713316 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" podUID="05704dba-5887-4069-a07f-6e1184ea6ee3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:21:21.711412 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:21.711370 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" podUID="05704dba-5887-4069-a07f-6e1184ea6ee3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:21:28.655033 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:21:28.654983 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:21:31.711705 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:31.711664 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" podUID="05704dba-5887-4069-a07f-6e1184ea6ee3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:21:41.711835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:41.711802 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" podUID="05704dba-5887-4069-a07f-6e1184ea6ee3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:21:46.093963 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.093932 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs"] Nov 03 00:21:46.095250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.095221 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" podUID="05704dba-5887-4069-a07f-6e1184ea6ee3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:21:46.165569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.165539 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk"] Nov 03 00:21:46.169241 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.169226 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.222261 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.222230 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f167aa83ac0d\"" Nov 03 00:21:46.222573 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.222555 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-088018b0f6be\"" Nov 03 00:21:46.227928 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.227905 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk"] Nov 03 00:21:46.231814 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.231789 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-plugins-config-volume\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.231932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.231832 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-provider-credentials\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.231932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.231872 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-routing-overlay-volume\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.231932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.231905 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m5qd\" (UniqueName: \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-kube-api-access-2m5qd\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.332571 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.332537 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-plugins-config-volume\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.332571 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.332571 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-provider-credentials\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.332816 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.332609 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-routing-overlay-volume\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.332816 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.332646 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m5qd\" (UniqueName: \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-kube-api-access-2m5qd\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.333481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.333424 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-routing-overlay-volume\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.333597 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.333495 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-plugins-config-volume\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.335270 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.335249 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-provider-credentials\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.390111 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.390050 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m5qd\" (UniqueName: \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-kube-api-access-2m5qd\") pod \"payload-processing-external-model-85c6f868cb-krhkk\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.482802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.482750 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.856805 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.856757 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk"] Nov 03 00:21:46.858336 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:21:46.858304 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a05409_b3ab_457e_9458_3e8da111e6fa.slice/crio-48492ef4fa6dd86afaa8e0a2b5e9c7bd1e008ea2d4bb596404c9163c1f7f9b8d WatchSource:0}: Error finding container 48492ef4fa6dd86afaa8e0a2b5e9c7bd1e008ea2d4bb596404c9163c1f7f9b8d: Status 404 returned error can't find the container with id 48492ef4fa6dd86afaa8e0a2b5e9c7bd1e008ea2d4bb596404c9163c1f7f9b8d Nov 03 00:21:46.983812 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.983758 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" event={"ID":"93a05409-b3ab-457e-9458-3e8da111e6fa","Type":"ContainerStarted","Data":"4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b"} Nov 03 00:21:46.984016 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.983847 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:46.984016 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:46.983873 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" event={"ID":"93a05409-b3ab-457e-9458-3e8da111e6fa","Type":"ContainerStarted","Data":"48492ef4fa6dd86afaa8e0a2b5e9c7bd1e008ea2d4bb596404c9163c1f7f9b8d"} Nov 03 00:21:47.098838 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:47.098754 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" podStartSLOduration=1.098741117 podStartE2EDuration="1.098741117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:47.096626078 +0000 UTC m=+990.380183753" watchObservedRunningTime="2026-11-03 00:21:47.098741117 +0000 UTC m=+990.382298790" Nov 03 00:21:51.094854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.094794 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" podUID="05704dba-5887-4069-a07f-6e1184ea6ee3" containerName="payload-processing" containerID="cri-o://c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15" gracePeriod=25 Nov 03 00:21:51.427935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.427894 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:21:51.485613 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.485575 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-provider-credentials\") pod \"05704dba-5887-4069-a07f-6e1184ea6ee3\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " Nov 03 00:21:51.485803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.485646 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-routing-overlay-volume\") pod \"05704dba-5887-4069-a07f-6e1184ea6ee3\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " Nov 03 00:21:51.485803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.485684 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-plugins-config-volume\") pod \"05704dba-5887-4069-a07f-6e1184ea6ee3\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " Nov 03 00:21:51.485803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.485754 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-kube-api-access-qvjk4\" (UniqueName: \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-kube-api-access-qvjk4\") pod \"05704dba-5887-4069-a07f-6e1184ea6ee3\" (UID: \"05704dba-5887-4069-a07f-6e1184ea6ee3\") " Nov 03 00:21:51.486603 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.486572 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-plugins-config-volume" pod "05704dba-5887-4069-a07f-6e1184ea6ee3" (UID: "05704dba-5887-4069-a07f-6e1184ea6ee3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:21:51.487041 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.487016 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-routing-overlay-volume" pod "05704dba-5887-4069-a07f-6e1184ea6ee3" (UID: "05704dba-5887-4069-a07f-6e1184ea6ee3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:21:51.488608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.488526 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-provider-credentials" pod "05704dba-5887-4069-a07f-6e1184ea6ee3" (UID: "05704dba-5887-4069-a07f-6e1184ea6ee3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:51.488717 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.488639 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-kube-api-access-qvjk4" pod "05704dba-5887-4069-a07f-6e1184ea6ee3" (UID: "05704dba-5887-4069-a07f-6e1184ea6ee3"). InnerVolumeSpecName "kube-api-access-qvjk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:51.587879 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.587832 4304 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-provider-credentials\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:51.588078 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.587887 4304 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-routing-overlay-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:51.588078 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.587904 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05704dba-5887-4069-a07f-6e1184ea6ee3-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:51.588078 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:51.587918 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvjk4\" (UniqueName: \"kubernetes.io/projected/05704dba-5887-4069-a07f-6e1184ea6ee3-kube-api-access-qvjk4\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:52.076814 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:52.067958 4304 generic.go:356] "Generic (PLEG): container finished" podID="05704dba-5887-4069-a07f-6e1184ea6ee3" containerID="c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15" exitCode=0 Nov 03 00:21:52.076814 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:52.068121 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" event={"ID":"05704dba-5887-4069-a07f-6e1184ea6ee3","Type":"ContainerDied","Data":"c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15"} Nov 03 00:21:52.076814 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:52.068177 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" event={"ID":"05704dba-5887-4069-a07f-6e1184ea6ee3","Type":"ContainerDied","Data":"db47b63c2d833dd0214424e7389e6d98fd97a2e40f94e3afe2eac5abe8d2f56b"} Nov 03 00:21:52.076814 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:52.068209 4304 scope.go:122] "RemoveContainer" containerID="c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15" Nov 03 00:21:52.076814 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:52.068381 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs" Nov 03 00:21:52.199534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:52.199490 4304 scope.go:122] "RemoveContainer" containerID="c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15" Nov 03 00:21:52.200711 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:21:52.200670 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15\": container with ID starting with c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15 not found: ID does not exist" containerID="c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15" Nov 03 00:21:52.200952 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:52.200911 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15"} err="failed to get container status \"c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15\": rpc error: code = NotFound desc = could not find container \"c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15\": container with ID starting with c95e90a11da72d04966d3c15f635d85b414dc8295a58731d76afd25db894bb15 not found: ID does not exist" Nov 03 00:21:52.287801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:52.276758 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs"] Nov 03 00:21:52.328859 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:52.325571 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-8qlgs"] Nov 03 00:21:52.868802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:52.867109 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05704dba-5887-4069-a07f-6e1184ea6ee3" path="/var/lib/kubelet/pods/05704dba-5887-4069-a07f-6e1184ea6ee3/volumes" Nov 03 00:21:57.627584 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.627539 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld"] Nov 03 00:21:57.628018 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.628005 4304 cpu_manager.go:414] "containerMap: removing container" podUID="05704dba-5887-4069-a07f-6e1184ea6ee3" containerName="payload-processing" Nov 03 00:21:57.628082 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.628019 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05704dba-5887-4069-a07f-6e1184ea6ee3" containerName="payload-processing" Nov 03 00:21:57.628133 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.628124 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="05704dba-5887-4069-a07f-6e1184ea6ee3" containerName="payload-processing" Nov 03 00:21:57.632672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.632648 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.817518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.817478 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.817705 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.817527 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32844841-6231-4493-92e4-46a7ff8a67bb-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.817705 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.817549 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.817705 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.817673 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.817851 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.817753 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.817851 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.817830 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.817926 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.817875 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32844841-6231-4493-92e4-46a7ff8a67bb-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.817985 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.817968 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7k8p\" (UniqueName: \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-kube-api-access-r7k8p\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.818031 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.818013 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.837432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.837403 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld"] Nov 03 00:21:57.918886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.918858 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.919053 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.919025 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32844841-6231-4493-92e4-46a7ff8a67bb-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.919125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.919063 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.919125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.919096 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.919240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.919184 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.919240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.919218 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.919342 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.919242 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32844841-6231-4493-92e4-46a7ff8a67bb-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.919342 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.919264 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.919673 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.919518 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7k8p\" (UniqueName: \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-kube-api-access-r7k8p\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.919874 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.919855 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.920024 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.919980 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.920792 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.920252 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.920792 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.920356 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.921129 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.921105 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32844841-6231-4493-92e4-46a7ff8a67bb-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.922608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.922576 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.924079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.923443 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32844841-6231-4493-92e4-46a7ff8a67bb-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.975712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.975682 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.975870 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.975843 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7k8p\" (UniqueName: \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-kube-api-access-r7k8p\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:57.992666 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:57.992645 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:21:58.245182 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:58.245103 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:58.654295 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:21:58.654250 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:21:58.689861 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:21:58.689838 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32844841_6231_4493_92e4_46a7ff8a67bb.slice/crio-69b391eaa64f13500a4780a8690053cbdc2ed2f12414e3883d884eadf93ea7cb WatchSource:0}: Error finding container 69b391eaa64f13500a4780a8690053cbdc2ed2f12414e3883d884eadf93ea7cb: Status 404 returned error can't find the container with id 69b391eaa64f13500a4780a8690053cbdc2ed2f12414e3883d884eadf93ea7cb Nov 03 00:21:58.692279 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:58.692245 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:58.692407 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:58.692317 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:58.692407 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:58.692350 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:58.695241 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:58.695219 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld"] Nov 03 00:21:59.156805 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:59.156745 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" event={"ID":"32844841-6231-4493-92e4-46a7ff8a67bb","Type":"ContainerStarted","Data":"b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd"} Nov 03 00:21:59.157014 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:59.156833 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" event={"ID":"32844841-6231-4493-92e4-46a7ff8a67bb","Type":"ContainerStarted","Data":"69b391eaa64f13500a4780a8690053cbdc2ed2f12414e3883d884eadf93ea7cb"} Nov 03 00:21:59.246818 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:59.245591 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:21:59.247606 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:59.247575 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:21:59.247733 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:59.247638 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 00:21:59.352463 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:21:59.352409 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" podStartSLOduration=2.352394565 podStartE2EDuration="2.352394565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:59.332392499 +0000 UTC m=+1002.615950213" watchObservedRunningTime="2026-11-03 00:21:59.352394565 +0000 UTC m=+1002.635952239" Nov 03 00:22:00.246431 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:00.246390 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:22:00.246888 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:00.246463 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 00:22:01.249725 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:01.249277 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:22:01.249725 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:01.249342 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 00:22:02.246242 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:02.246203 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:22:02.246465 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:02.246284 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 00:22:03.246254 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:03.246214 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:22:03.246673 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:03.246310 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 00:22:04.246273 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:04.246235 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:22:04.246670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:04.246291 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 00:22:05.245647 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:05.245611 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 00:22:05.245840 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:05.245684 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 00:22:06.250520 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:06.250491 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:22:06.251050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:06.251037 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:22:06.251746 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:06.251725 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:22:28.654652 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:22:28.654612 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:22:29.323336 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.323259 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz"] Nov 03 00:22:29.328209 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.328193 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.400019 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.399990 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e28e1a\"/\"kube-root-ca.crt\"" Nov 03 00:22:29.430172 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.430139 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e28e1a\"/\"openshift-service-ca.crt\"" Nov 03 00:22:29.430390 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.430370 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e28e1a\"/\"default-dockercfg-7v5bh\"" Nov 03 00:22:29.430482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.430395 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e28e1a\"/\"distinct-w1-e28e1a-kserve-self-signed-certs\"" Nov 03 00:22:29.431847 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.431824 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-dshm\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.431920 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.431867 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-tmp-dir\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.431965 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.431954 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb1752b-050f-4042-8650-0be7a24c004e-tls-certs\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.432004 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.431995 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtdzs\" (UniqueName: \"kubernetes.io/projected/2fb1752b-050f-4042-8650-0be7a24c004e-kube-api-access-wtdzs\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.432111 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.432086 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-model-cache\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.432155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.432133 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-home\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.433187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.433170 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz"] Nov 03 00:22:29.533303 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.533266 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-model-cache\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.533507 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.533316 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-home\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.533507 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.533406 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-dshm\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.533507 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.533445 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-tmp-dir\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.533699 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.533591 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb1752b-050f-4042-8650-0be7a24c004e-tls-certs\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.533699 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.533638 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtdzs\" (UniqueName: \"kubernetes.io/projected/2fb1752b-050f-4042-8650-0be7a24c004e-kube-api-access-wtdzs\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.533699 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.533666 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-model-cache\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.533884 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.533731 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-home\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.533884 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.533806 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-tmp-dir\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.535822 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.535800 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-dshm\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.536076 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.536060 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb1752b-050f-4042-8650-0be7a24c004e-tls-certs\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.590648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.590584 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtdzs\" (UniqueName: \"kubernetes.io/projected/2fb1752b-050f-4042-8650-0be7a24c004e-kube-api-access-wtdzs\") pod \"distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:29.640960 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:29.640931 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:30.021909 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:30.021884 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz"] Nov 03 00:22:30.023199 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:22:30.023170 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fb1752b_050f_4042_8650_0be7a24c004e.slice/crio-f365b4d22a25c802b8d181760a68b5b7369356d10f975390310defdbcfb9ebb4 WatchSource:0}: Error finding container f365b4d22a25c802b8d181760a68b5b7369356d10f975390310defdbcfb9ebb4: Status 404 returned error can't find the container with id f365b4d22a25c802b8d181760a68b5b7369356d10f975390310defdbcfb9ebb4 Nov 03 00:22:30.386505 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:30.386418 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" event={"ID":"2fb1752b-050f-4042-8650-0be7a24c004e","Type":"ContainerStarted","Data":"06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef"} Nov 03 00:22:30.386505 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:30.386466 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" event={"ID":"2fb1752b-050f-4042-8650-0be7a24c004e","Type":"ContainerStarted","Data":"f365b4d22a25c802b8d181760a68b5b7369356d10f975390310defdbcfb9ebb4"} Nov 03 00:22:30.502832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:30.502761 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" podStartSLOduration=1.50274754 podStartE2EDuration="1.50274754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:30.500124184 +0000 UTC m=+1033.783681860" watchObservedRunningTime="2026-11-03 00:22:30.50274754 +0000 UTC m=+1033.786305211" Nov 03 00:22:30.641681 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:30.641573 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:39.641066 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:39.641028 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:39.653594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:39.653570 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:40.469105 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:22:40.469073 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:22:58.654235 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:22:58.654198 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:23:28.313178 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.313143 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58"] Nov 03 00:23:28.317099 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.317080 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.423434 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.423400 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.423633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.423473 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phq4t\" (UniqueName: \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-kube-api-access-phq4t\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.423633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.423554 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.423633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.423603 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.423854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.423652 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.423854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.423695 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.423854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.423768 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.424014 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.423886 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.424014 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.423933 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.425150 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.425121 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58"] Nov 03 00:23:28.525428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.525389 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.525622 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.525449 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.525622 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.525472 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.525622 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.525509 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phq4t\" (UniqueName: \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-kube-api-access-phq4t\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.525622 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.525533 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.525622 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.525579 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.525943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.525748 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.525943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.525809 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.525943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.525848 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.526089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.526002 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.526089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.526046 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.526179 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.526100 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.526274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.526254 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.526338 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.526264 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.528105 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.528087 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.528161 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.528138 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.582322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.582240 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.582457 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.582338 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phq4t\" (UniqueName: \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-kube-api-access-phq4t\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.630224 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:28.630191 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:28.721051 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:23:28.721014 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:23:29.108517 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:29.108460 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58"] Nov 03 00:23:29.162579 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:23:29.162544 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2cf84f_1a22_49da_80a3_56ce09040f1c.slice/crio-7a4d00ebfc30f01f2a57fa0c5998a212e86a1c9cf0cbcfe7370b894a322a1573 WatchSource:0}: Error finding container 7a4d00ebfc30f01f2a57fa0c5998a212e86a1c9cf0cbcfe7370b894a322a1573: Status 404 returned error can't find the container with id 7a4d00ebfc30f01f2a57fa0c5998a212e86a1c9cf0cbcfe7370b894a322a1573 Nov 03 00:23:29.166119 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:29.166067 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:29.166349 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:29.166329 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:29.166516 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:29.166490 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:29.751801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:29.741401 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" event={"ID":"9b2cf84f-1a22-49da-80a3-56ce09040f1c","Type":"ContainerStarted","Data":"10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824"} Nov 03 00:23:29.751801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:29.741472 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" event={"ID":"9b2cf84f-1a22-49da-80a3-56ce09040f1c","Type":"ContainerStarted","Data":"7a4d00ebfc30f01f2a57fa0c5998a212e86a1c9cf0cbcfe7370b894a322a1573"} Nov 03 00:23:29.866801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:29.865967 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" podStartSLOduration=1.8659455619999998 podStartE2EDuration="1.865945562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:29.857269919 +0000 UTC m=+1093.140827609" watchObservedRunningTime="2026-11-03 00:23:29.865945562 +0000 UTC m=+1093.149503237" Nov 03 00:23:30.642799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:30.640964 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:30.642799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:30.641098 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 00:23:30.642799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:30.641152 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" podUID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 00:23:31.639683 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:31.639188 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 00:23:31.639683 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:31.639253 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" podUID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 00:23:32.644799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:32.643902 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 00:23:32.644799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:32.643972 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" podUID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 00:23:33.644800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:33.641225 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 00:23:33.644800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:33.641292 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" podUID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 00:23:34.633221 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:34.633179 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 00:23:34.633533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:34.633499 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" podUID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 00:23:35.630742 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:35.630706 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 00:23:35.631175 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:35.630765 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" podUID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 00:23:36.634830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:36.634798 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:36.635240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:36.635102 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:36.636015 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:23:36.635996 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:23:58.655022 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:23:58.654975 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:24:08.454614 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.454578 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8"] Nov 03 00:24:08.458292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.458268 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.556331 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.556302 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.556510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.556360 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.556510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.556485 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.556612 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.556542 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.556612 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.556568 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05faa1fc-ab4e-49ba-935c-27de58df56f1-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.556715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.556685 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.556865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.556841 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.556975 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.556959 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvg5\" (UniqueName: \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-kube-api-access-qnvg5\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.557049 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.557028 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.564510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.564487 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8"] Nov 03 00:24:08.657931 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.657897 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.657951 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658004 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvg5\" (UniqueName: \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-kube-api-access-qnvg5\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658029 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658056 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658090 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658115 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658441 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658147 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658441 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658181 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05faa1fc-ab4e-49ba-935c-27de58df56f1-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658554 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658493 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658554 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658523 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658664 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658619 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658718 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.658942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.658923 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05faa1fc-ab4e-49ba-935c-27de58df56f1-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.660375 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.660348 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.660975 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.660955 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.717462 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.717397 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.717584 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.717545 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvg5\" (UniqueName: \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-kube-api-access-qnvg5\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-zg8r8\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:08.771982 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:08.771950 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:09.197823 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:09.197791 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8"] Nov 03 00:24:09.198659 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:24:09.198633 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05faa1fc_ab4e_49ba_935c_27de58df56f1.slice/crio-968dfe7387119cb71a8ce8967233416e5d4a924ac318602796c871b0617f86e0 WatchSource:0}: Error finding container 968dfe7387119cb71a8ce8967233416e5d4a924ac318602796c871b0617f86e0: Status 404 returned error can't find the container with id 968dfe7387119cb71a8ce8967233416e5d4a924ac318602796c871b0617f86e0 Nov 03 00:24:09.201037 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:09.201004 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:09.201115 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:09.201069 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:09.201115 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:09.201097 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:09.980426 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:09.980386 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" event={"ID":"05faa1fc-ab4e-49ba-935c-27de58df56f1","Type":"ContainerStarted","Data":"a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198"} Nov 03 00:24:09.980886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:09.980450 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" event={"ID":"05faa1fc-ab4e-49ba-935c-27de58df56f1","Type":"ContainerStarted","Data":"968dfe7387119cb71a8ce8967233416e5d4a924ac318602796c871b0617f86e0"} Nov 03 00:24:10.204233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:10.204169 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" podStartSLOduration=2.204149995 podStartE2EDuration="2.204149995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:10.152134553 +0000 UTC m=+1133.435692229" watchObservedRunningTime="2026-11-03 00:24:10.204149995 +0000 UTC m=+1133.487707670" Nov 03 00:24:10.772364 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:10.772330 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:10.774222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:10.774197 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:24:10.774354 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:10.774248 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 00:24:11.772646 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:11.772605 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:24:11.773186 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:11.772670 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 00:24:12.772603 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:12.772554 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:24:12.772840 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:12.772640 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 00:24:13.773345 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:13.773306 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:24:13.774048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:13.773999 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 00:24:14.492744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:14.492686 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" podUID="2fb1752b-050f-4042-8650-0be7a24c004e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:14.773275 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:14.773176 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:24:14.773275 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:14.773237 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 00:24:15.772718 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:15.772682 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:24:15.772936 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:15.772749 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 00:24:17.813932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:17.813887 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:24:17.814437 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:17.813953 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:17.960924 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:17.960896 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:17.961295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:17.961276 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:17.962025 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:17.962004 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:24:20.885417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.885383 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk"] Nov 03 00:24:20.889566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.889547 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:20.941444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.941415 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e28e1a-openshift-default-dockercfg-4zwkf\"" Nov 03 00:24:20.941634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.941611 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:20.941702 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.941645 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:20.941751 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.941696 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea622cbd-cfbc-44d5-81be-d412de79058e-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:20.941930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.941908 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:20.941988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.941968 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:20.942063 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.942043 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:20.942202 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.942187 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vh2d\" (UniqueName: \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-kube-api-access-4vh2d\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:20.942272 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.942257 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:20.942313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.942291 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:20.993962 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:20.993930 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk"] Nov 03 00:24:21.044237 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.044176 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.044996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.044970 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.045549 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.045527 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.045728 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.045560 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.045728 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.045611 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea622cbd-cfbc-44d5-81be-d412de79058e-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.045728 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.045700 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.046088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.045753 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.046088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.045803 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.046088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.045948 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vh2d\" (UniqueName: \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-kube-api-access-4vh2d\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.046349 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.045482 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.047286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.047260 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.047596 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.047575 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea622cbd-cfbc-44d5-81be-d412de79058e-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.049022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.048988 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.049206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.049189 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.049872 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.049850 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.050238 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.050214 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.103151 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.103121 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vh2d\" (UniqueName: \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-kube-api-access-4vh2d\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.103321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.103286 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.203300 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.203262 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:21.653821 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:24:21.653786 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea622cbd_cfbc_44d5_81be_d412de79058e.slice/crio-2400b3c462fb49897e67985df11be87bb8043cd3a2101a603e7809d423d10438 WatchSource:0}: Error finding container 2400b3c462fb49897e67985df11be87bb8043cd3a2101a603e7809d423d10438: Status 404 returned error can't find the container with id 2400b3c462fb49897e67985df11be87bb8043cd3a2101a603e7809d423d10438 Nov 03 00:24:21.655586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.655562 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk"] Nov 03 00:24:21.656712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.656676 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:21.656812 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.656757 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:21.656861 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:21.656826 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:22.093147 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:22.093053 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" event={"ID":"ea622cbd-cfbc-44d5-81be-d412de79058e","Type":"ContainerStarted","Data":"77c13092cebbc23c4383bdfba80568f23dbf7847a04454d80d2efd1cd00bad0f"} Nov 03 00:24:22.093147 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:22.093116 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" event={"ID":"ea622cbd-cfbc-44d5-81be-d412de79058e","Type":"ContainerStarted","Data":"2400b3c462fb49897e67985df11be87bb8043cd3a2101a603e7809d423d10438"} Nov 03 00:24:22.204318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:22.204294 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:22.205659 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:22.205638 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 00:24:22.205764 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:22.205680 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 00:24:22.211030 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:22.210991 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" podStartSLOduration=2.210978018 podStartE2EDuration="2.210978018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:22.208503956 +0000 UTC m=+1145.492061668" watchObservedRunningTime="2026-11-03 00:24:22.210978018 +0000 UTC m=+1145.494535692" Nov 03 00:24:23.204362 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:23.204326 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 00:24:23.204744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:23.204379 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 00:24:24.190139 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.190105 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6"] Nov 03 00:24:24.194091 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.194076 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.204260 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.204237 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 00:24:24.204396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.204286 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 00:24:24.288637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.285611 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.288637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.285668 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb2a3f42-c317-4d84-9941-c12df6a7c752-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.288637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.285714 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.288637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.285804 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.288637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.285853 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.288637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.285873 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.288637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.285899 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzhc\" (UniqueName: \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-kube-api-access-hvzhc\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.288637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.285950 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.288637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.285969 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.298959 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.298927 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6"] Nov 03 00:24:24.387126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387086 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387134 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb2a3f42-c317-4d84-9941-c12df6a7c752-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387179 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387243 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387293 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387322 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387363 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzhc\" (UniqueName: \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-kube-api-access-hvzhc\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387427 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387463 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387501 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387587 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387887 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387820 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.387978 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387958 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb2a3f42-c317-4d84-9941-c12df6a7c752-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.388031 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.387960 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.389809 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.389762 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.390038 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.390017 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.447754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.447680 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzhc\" (UniqueName: \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-kube-api-access-hvzhc\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.447922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.447760 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.507588 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.507562 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:24.963156 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.963129 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6"] Nov 03 00:24:24.964020 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:24:24.963990 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2a3f42_c317_4d84_9941_c12df6a7c752.slice/crio-69db29582465ad5eb1d92b6cb64d3b3613549fe97b6e41bc665e36aa5cedbe0d WatchSource:0}: Error finding container 69db29582465ad5eb1d92b6cb64d3b3613549fe97b6e41bc665e36aa5cedbe0d: Status 404 returned error can't find the container with id 69db29582465ad5eb1d92b6cb64d3b3613549fe97b6e41bc665e36aa5cedbe0d Nov 03 00:24:24.966508 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.966480 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:24.966579 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.966541 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:24.966613 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:24.966589 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:25.113149 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:25.113113 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" event={"ID":"cb2a3f42-c317-4d84-9941-c12df6a7c752","Type":"ContainerStarted","Data":"e6043b07c2515d1fded97c9db7363494888b2757d471065bf6867e234276bed4"} Nov 03 00:24:25.113322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:25.113162 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" event={"ID":"cb2a3f42-c317-4d84-9941-c12df6a7c752","Type":"ContainerStarted","Data":"69db29582465ad5eb1d92b6cb64d3b3613549fe97b6e41bc665e36aa5cedbe0d"} Nov 03 00:24:25.214424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:25.205385 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 00:24:25.214424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:25.205440 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 00:24:25.242801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:25.231181 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" podStartSLOduration=1.231160087 podStartE2EDuration="1.231160087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:25.228228557 +0000 UTC m=+1148.511786234" watchObservedRunningTime="2026-11-03 00:24:25.231160087 +0000 UTC m=+1148.514717760" Nov 03 00:24:25.508079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:25.508001 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:26.207803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:26.207422 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 00:24:26.207803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:26.207478 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 00:24:26.521861 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:26.510430 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:24:26.521861 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:26.510492 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 00:24:27.217429 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:27.217381 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 00:24:27.217650 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:27.217452 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 00:24:27.533856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:27.532859 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:24:27.533856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:27.532924 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 00:24:27.553825 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:27.549001 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:24:27.553825 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:27.549085 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:27.789116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:27.789017 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" podUID="aa4f830f-1dc3-4231-ab46-49e522d2e794" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 00:24:28.203922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:28.203890 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 00:24:28.204104 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:28.203940 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 00:24:28.508141 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:28.508060 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:24:28.508141 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:28.508120 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 00:24:28.672288 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:24:28.672252 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:24:29.508583 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:29.508545 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:24:29.508791 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:29.508611 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 00:24:30.205055 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:30.205009 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:24:30.205534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:30.205071 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:30.508553 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:30.508468 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:24:30.508553 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:30.508525 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 00:24:31.053797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:31.053314 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:31.054274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:31.054220 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:31.056313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:31.056283 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:24:31.508617 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:31.508584 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:24:31.509210 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:31.509180 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 00:24:32.509002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:32.508963 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:24:32.509402 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:32.509032 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 00:24:33.512681 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:33.512645 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:33.513143 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:33.513079 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:33.513738 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:33.513720 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:24:39.310733 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.310701 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5"] Nov 03 00:24:39.314162 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.314144 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.392753 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.392672 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.392977 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.392845 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.392977 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.392872 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvktn\" (UniqueName: \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-kube-api-access-pvktn\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.392977 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.392931 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.393159 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.393029 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.393159 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.393081 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.393279 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.393173 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.393279 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.393266 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.393381 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.393295 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.408642 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.408608 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5"] Nov 03 00:24:39.494958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.494930 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.494958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.494959 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495182 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495091 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495234 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495177 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495234 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495203 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvktn\" (UniqueName: \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-kube-api-access-pvktn\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495262 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495284 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495315 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495372 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495426 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495366 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495705 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495546 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495705 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495651 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.495822 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.495740 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.497674 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.497655 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.498023 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.498004 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.552113 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.552080 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.552277 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.552149 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvktn\" (UniqueName: \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-kube-api-access-pvktn\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:39.627564 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:39.627493 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:40.056951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:40.056923 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5"] Nov 03 00:24:40.057369 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:24:40.057347 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd63ece_35a8_4ac8_8678_9c6ff2b2f200.slice/crio-aa841950e472456721b4ab9d94ce0683d8739f30dc47738cea126205729cc0b8 WatchSource:0}: Error finding container aa841950e472456721b4ab9d94ce0683d8739f30dc47738cea126205729cc0b8: Status 404 returned error can't find the container with id aa841950e472456721b4ab9d94ce0683d8739f30dc47738cea126205729cc0b8 Nov 03 00:24:40.059719 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:40.059688 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:40.059815 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:40.059753 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:40.059859 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:40.059813 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:40.338560 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:40.338476 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" event={"ID":"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200","Type":"ContainerStarted","Data":"f0e826d3b299db82a6ba5e86303ed6d33e1c4c0d42e0f6c30a1458fb49b5b4dd"} Nov 03 00:24:40.338560 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:40.338527 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" event={"ID":"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200","Type":"ContainerStarted","Data":"aa841950e472456721b4ab9d94ce0683d8739f30dc47738cea126205729cc0b8"} Nov 03 00:24:40.455942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:40.455884 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" podStartSLOduration=1.455864582 podStartE2EDuration="1.455864582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:40.454436017 +0000 UTC m=+1163.737993728" watchObservedRunningTime="2026-11-03 00:24:40.455864582 +0000 UTC m=+1163.739422255" Nov 03 00:24:40.628008 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:40.627898 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:41.643801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:41.636055 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:24:41.643801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:41.636116 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 00:24:42.632300 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:42.632262 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:24:42.632590 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:42.632560 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 00:24:43.646808 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:43.646751 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:24:43.647259 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:43.646841 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 00:24:44.641802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:44.638179 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:24:44.641802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:44.638254 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 00:24:45.634209 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:45.634172 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:24:45.634844 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:45.634809 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 00:24:46.628563 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:46.628527 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 00:24:46.628738 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:46.628588 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 00:24:47.632545 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:47.632504 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:47.632962 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:47.632896 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:47.633651 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:47.633629 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:24:51.463928 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:51.463873 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" podUID="2fb1752b-050f-4042-8650-0be7a24c004e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:51.532852 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:51.525906 4304 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" podUID="49aef8b0-ab85-4ccc-bb3d-c5eebfa27469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:51.754582 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:51.754457 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" podUID="082c2451-3bb7-4b3a-82c0-f8ea634adcdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:51.775925 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:51.775862 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" podUID="aa4f830f-1dc3-4231-ab46-49e522d2e794" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 00:24:51.910150 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:51.910095 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" podUID="26d57814-f91a-423c-ab38-5cb55e4d77a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.300923 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:52.300878 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:24:52.301139 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:52.300950 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.465082 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:52.465028 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" podUID="2fb1752b-050f-4042-8650-0be7a24c004e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.466347 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:52.466324 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:24:52.487934 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:52.487906 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 00:24:52.768810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:52.764975 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" podUID="082c2451-3bb7-4b3a-82c0-f8ea634adcdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.796948 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:52.796917 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:24:52.801808 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:52.798499 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" podUID="aa4f830f-1dc3-4231-ab46-49e522d2e794" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.820278 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:52.819408 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:24:53.053583 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:53.053498 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 00:24:53.187220 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:24:53.187190 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 00:24:58.655884 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:24:58.655848 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:25:08.924122 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:08.924088 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw"] Nov 03 00:25:08.927986 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:08.927967 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:08.990038 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:08.990005 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jfw\" (UniqueName: \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-kube-api-access-28jfw\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:08.990241 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:08.990138 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6794e0ff-4c6d-48d0-9982-6241c13b216d-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:08.990241 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:08.990164 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:08.990241 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:08.990216 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:08.990423 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:08.990304 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:08.990423 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:08.990330 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:08.990423 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:08.990367 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:08.990423 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:08.990390 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:08.990423 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:08.990413 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.032545 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.032512 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw"] Nov 03 00:25:09.091722 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.091691 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.091943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.091735 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.091943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.091804 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.091943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.091828 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.091943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.091851 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.091943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.091909 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jfw\" (UniqueName: \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-kube-api-access-28jfw\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.092229 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.091994 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6794e0ff-4c6d-48d0-9982-6241c13b216d-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.092229 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.092020 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.092229 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.092089 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.092378 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.092287 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.092378 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.092311 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.092604 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.092471 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.092604 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.092526 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.092754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.092640 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6794e0ff-4c6d-48d0-9982-6241c13b216d-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.094344 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.094318 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.094468 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.094362 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.148644 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.148607 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jfw\" (UniqueName: \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-kube-api-access-28jfw\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.148644 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.148619 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-j7lgw\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.242023 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.241945 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:09.675316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.675287 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw"] Nov 03 00:25:09.676472 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:25:09.676440 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6794e0ff_4c6d_48d0_9982_6241c13b216d.slice/crio-4deb2461391afada4d9c2677a9da2a240b279c5cb149a41acce652284b3aaafb WatchSource:0}: Error finding container 4deb2461391afada4d9c2677a9da2a240b279c5cb149a41acce652284b3aaafb: Status 404 returned error can't find the container with id 4deb2461391afada4d9c2677a9da2a240b279c5cb149a41acce652284b3aaafb Nov 03 00:25:09.679340 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.679305 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:09.679420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.679389 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:09.679460 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:09.679429 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:10.566177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:10.566138 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" event={"ID":"6794e0ff-4c6d-48d0-9982-6241c13b216d","Type":"ContainerStarted","Data":"c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f"} Nov 03 00:25:10.566655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:10.566217 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" event={"ID":"6794e0ff-4c6d-48d0-9982-6241c13b216d","Type":"ContainerStarted","Data":"4deb2461391afada4d9c2677a9da2a240b279c5cb149a41acce652284b3aaafb"} Nov 03 00:25:10.745079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:10.745009 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" podStartSLOduration=2.744973669 podStartE2EDuration="2.744973669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:10.743515009 +0000 UTC m=+1194.027072683" watchObservedRunningTime="2026-11-03 00:25:10.744973669 +0000 UTC m=+1194.028531345" Nov 03 00:25:11.243802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:11.243155 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:11.247676 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:11.247636 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 00:25:11.247902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:11.247697 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 00:25:12.246324 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:12.246283 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 00:25:12.246919 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:12.246887 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 00:25:13.246221 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.246178 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 00:25:13.246401 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.246240 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 00:25:13.614399 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.614315 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7"] Nov 03 00:25:13.618288 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.618272 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.770899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.770866 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.770899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.770908 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.771153 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.771126 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvjf8\" (UniqueName: \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-kube-api-access-vvjf8\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.771195 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.771177 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.771265 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.771250 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.771363 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.771308 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.771363 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.771335 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.771456 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.771366 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.771556 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.771537 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.856258 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.856219 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7"] Nov 03 00:25:13.872827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.872729 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.872980 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.872864 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.872980 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.872911 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.873106 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.873007 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvjf8\" (UniqueName: \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-kube-api-access-vvjf8\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.873106 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.873045 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.873106 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.873085 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.873277 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.873139 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.873277 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.873167 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.873277 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.873199 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.873429 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.873273 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.873592 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.873564 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.873676 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.873583 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.873676 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.873662 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.873914 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.873892 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.876662 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.876639 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.876916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.876896 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.948370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.948316 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.948559 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.948450 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvjf8\" (UniqueName: \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-kube-api-access-vvjf8\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:13.955498 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:13.955471 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:14.245678 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:14.245178 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 00:25:14.245678 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:14.245258 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 00:25:14.544859 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:14.543214 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7"] Nov 03 00:25:14.558901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:14.550054 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:14.558901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:14.550140 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:14.558901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:14.550189 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:14.595580 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:14.595546 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" event={"ID":"31b6c870-bf3c-4bd1-8ffb-acf522c710d9","Type":"ContainerStarted","Data":"0b289fd2eba0b8337abdee78185c811ad3467d7d02516c051c1fccf0a98cd9de"} Nov 03 00:25:15.245795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:15.245738 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 00:25:15.246114 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:15.246068 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 00:25:15.619857 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:15.619626 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" event={"ID":"31b6c870-bf3c-4bd1-8ffb-acf522c710d9","Type":"ContainerStarted","Data":"257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f"} Nov 03 00:25:15.847800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:15.847487 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" podStartSLOduration=2.847466754 podStartE2EDuration="2.847466754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:15.792329441 +0000 UTC m=+1199.075887115" watchObservedRunningTime="2026-11-03 00:25:15.847466754 +0000 UTC m=+1199.131024430" Nov 03 00:25:15.958464 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:15.958364 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:15.960380 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:15.960349 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 00:25:15.960615 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:15.960588 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 00:25:16.261858 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:16.251344 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 00:25:16.261858 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:16.251402 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 00:25:16.970908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:16.970790 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 00:25:16.999283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:16.998917 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 00:25:17.008311 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.005972 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n"] Nov 03 00:25:17.010818 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.010793 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n"] Nov 03 00:25:17.010930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.010882 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:25:17.064186 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.064156 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cb65f3a5\"" Nov 03 00:25:17.064337 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.064157 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cb65f3a5-dockercfg-5jsvb\"" Nov 03 00:25:17.104317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.104277 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2s8f\" (UniqueName: \"kubernetes.io/projected/c69b9a91-3d04-430c-9ce1-58444ca13c00-kube-api-access-g2s8f\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n\" (UID: \"c69b9a91-3d04-430c-9ce1-58444ca13c00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:25:17.104480 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.104331 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69b9a91-3d04-430c-9ce1-58444ca13c00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n\" (UID: \"c69b9a91-3d04-430c-9ce1-58444ca13c00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:25:17.205546 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.205492 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2s8f\" (UniqueName: \"kubernetes.io/projected/c69b9a91-3d04-430c-9ce1-58444ca13c00-kube-api-access-g2s8f\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n\" (UID: \"c69b9a91-3d04-430c-9ce1-58444ca13c00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:25:17.206344 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.206320 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69b9a91-3d04-430c-9ce1-58444ca13c00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n\" (UID: \"c69b9a91-3d04-430c-9ce1-58444ca13c00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:25:17.207560 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.207530 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69b9a91-3d04-430c-9ce1-58444ca13c00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n\" (UID: \"c69b9a91-3d04-430c-9ce1-58444ca13c00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:25:17.261680 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.261591 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8"] Nov 03 00:25:17.267620 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.267595 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:25:17.282019 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.281811 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2s8f\" (UniqueName: \"kubernetes.io/projected/c69b9a91-3d04-430c-9ce1-58444ca13c00-kube-api-access-g2s8f\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n\" (UID: \"c69b9a91-3d04-430c-9ce1-58444ca13c00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:25:17.303508 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.303473 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8"] Nov 03 00:25:17.366713 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.362026 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:25:17.410800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.410644 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea296694-ea86-47cb-8e07-9470e09e1755-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8\" (UID: \"ea296694-ea86-47cb-8e07-9470e09e1755\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:25:17.411139 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.411083 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfjkj\" (UniqueName: \"kubernetes.io/projected/ea296694-ea86-47cb-8e07-9470e09e1755-kube-api-access-zfjkj\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8\" (UID: \"ea296694-ea86-47cb-8e07-9470e09e1755\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:25:17.553000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.515185 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfjkj\" (UniqueName: \"kubernetes.io/projected/ea296694-ea86-47cb-8e07-9470e09e1755-kube-api-access-zfjkj\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8\" (UID: \"ea296694-ea86-47cb-8e07-9470e09e1755\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:25:17.553000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.515376 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea296694-ea86-47cb-8e07-9470e09e1755-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8\" (UID: \"ea296694-ea86-47cb-8e07-9470e09e1755\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:25:17.553000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.519072 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea296694-ea86-47cb-8e07-9470e09e1755-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8\" (UID: \"ea296694-ea86-47cb-8e07-9470e09e1755\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:25:17.581808 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.579089 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfjkj\" (UniqueName: \"kubernetes.io/projected/ea296694-ea86-47cb-8e07-9470e09e1755-kube-api-access-zfjkj\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8\" (UID: \"ea296694-ea86-47cb-8e07-9470e09e1755\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:25:17.607408 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.606682 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:25:17.855356 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.855267 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n"] Nov 03 00:25:17.861684 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:25:17.861652 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69b9a91_3d04_430c_9ce1_58444ca13c00.slice/crio-fc18c74dc4aaefa16f22b3ccadf5188a776d5ff8b3024240ac38956de189ca97 WatchSource:0}: Error finding container fc18c74dc4aaefa16f22b3ccadf5188a776d5ff8b3024240ac38956de189ca97: Status 404 returned error can't find the container with id fc18c74dc4aaefa16f22b3ccadf5188a776d5ff8b3024240ac38956de189ca97 Nov 03 00:25:17.921112 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.920116 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:17.921983 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.921960 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:17.924600 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.924550 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:25:17.959416 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.956652 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 00:25:17.959416 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:17.956733 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 00:25:18.074026 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:18.074003 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8"] Nov 03 00:25:18.075001 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:25:18.074980 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea296694_ea86_47cb_8e07_9470e09e1755.slice/crio-927413eacab0a5611b3c9743e3402305fa86039febe72a00ac264b4bd2b9e58e WatchSource:0}: Error finding container 927413eacab0a5611b3c9743e3402305fa86039febe72a00ac264b4bd2b9e58e: Status 404 returned error can't find the container with id 927413eacab0a5611b3c9743e3402305fa86039febe72a00ac264b4bd2b9e58e Nov 03 00:25:18.682947 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:18.682910 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" event={"ID":"c69b9a91-3d04-430c-9ce1-58444ca13c00","Type":"ContainerStarted","Data":"627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb"} Nov 03 00:25:18.683117 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:18.682970 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" event={"ID":"c69b9a91-3d04-430c-9ce1-58444ca13c00","Type":"ContainerStarted","Data":"fc18c74dc4aaefa16f22b3ccadf5188a776d5ff8b3024240ac38956de189ca97"} Nov 03 00:25:18.683117 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:18.683020 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:25:18.684641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:18.684614 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" event={"ID":"ea296694-ea86-47cb-8e07-9470e09e1755","Type":"ContainerStarted","Data":"18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17"} Nov 03 00:25:18.684791 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:18.684666 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" event={"ID":"ea296694-ea86-47cb-8e07-9470e09e1755","Type":"ContainerStarted","Data":"927413eacab0a5611b3c9743e3402305fa86039febe72a00ac264b4bd2b9e58e"} Nov 03 00:25:18.684859 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:18.684812 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:25:18.807383 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:18.807316 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" podStartSLOduration=2.807299269 podStartE2EDuration="2.807299269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:18.806119533 +0000 UTC m=+1202.089677233" watchObservedRunningTime="2026-11-03 00:25:18.807299269 +0000 UTC m=+1202.090856943" Nov 03 00:25:18.941872 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:18.940324 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" podStartSLOduration=1.940299628 podStartE2EDuration="1.940299628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:18.918475073 +0000 UTC m=+1202.202032746" watchObservedRunningTime="2026-11-03 00:25:18.940299628 +0000 UTC m=+1202.223857303" Nov 03 00:25:18.971803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:18.966654 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 00:25:18.971803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:18.966731 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 00:25:19.956263 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:19.956223 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 00:25:19.956749 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:19.956280 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 00:25:20.956028 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:20.955985 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 00:25:20.956203 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:20.956049 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 00:25:21.392694 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.392606 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r"] Nov 03 00:25:21.396897 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.396878 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.503821 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.503788 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r"] Nov 03 00:25:21.506540 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.506508 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.506676 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.506590 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvqp2\" (UniqueName: \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-kube-api-access-rvqp2\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.506808 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.506756 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.506886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.506838 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.506886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.506868 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.506990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.506945 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.507044 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.507007 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.507102 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.507051 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.507154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.507110 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a79e336-f4c4-4d28-8023-be246046f3dc-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608005 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608059 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608194 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608369 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608291 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608369 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608349 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608399 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608446 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a79e336-f4c4-4d28-8023-be246046f3dc-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608600 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608491 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608600 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608533 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608600 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608542 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvqp2\" (UniqueName: \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-kube-api-access-rvqp2\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608747 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608610 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608836 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608816 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.608896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.608816 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.609415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.609391 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a79e336-f4c4-4d28-8023-be246046f3dc-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.610921 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.610898 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.610999 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.610929 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.664337 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.664313 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.664540 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.664519 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvqp2\" (UniqueName: \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-kube-api-access-rvqp2\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.714393 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.714360 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:21.956494 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.956418 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 00:25:21.956650 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:21.956495 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 00:25:22.172060 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:22.172016 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r"] Nov 03 00:25:22.209080 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:25:22.209005 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a79e336_f4c4_4d28_8023_be246046f3dc.slice/crio-0e36c998f7d521455a222dbd43ac27cf8838b939ccf7d0750cd338ebe79c9fed WatchSource:0}: Error finding container 0e36c998f7d521455a222dbd43ac27cf8838b939ccf7d0750cd338ebe79c9fed: Status 404 returned error can't find the container with id 0e36c998f7d521455a222dbd43ac27cf8838b939ccf7d0750cd338ebe79c9fed Nov 03 00:25:22.211634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:22.211599 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:22.211744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:22.211679 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:22.211744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:22.211709 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:22.718697 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:22.718651 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" event={"ID":"3a79e336-f4c4-4d28-8023-be246046f3dc","Type":"ContainerStarted","Data":"ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84"} Nov 03 00:25:22.719270 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:22.718737 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" event={"ID":"3a79e336-f4c4-4d28-8023-be246046f3dc","Type":"ContainerStarted","Data":"0e36c998f7d521455a222dbd43ac27cf8838b939ccf7d0750cd338ebe79c9fed"} Nov 03 00:25:22.846854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:22.846728 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" podStartSLOduration=1.8467100300000001 podStartE2EDuration="1.84671003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:22.846591277 +0000 UTC m=+1206.130148952" watchObservedRunningTime="2026-11-03 00:25:22.84671003 +0000 UTC m=+1206.130267705" Nov 03 00:25:22.961592 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:22.961556 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:22.962053 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:22.962023 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:22.963039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:22.963019 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:25:23.715012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:23.714977 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:23.716435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:23.716414 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:25:23.716550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:23.716464 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" podUID="3a79e336-f4c4-4d28-8023-be246046f3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 00:25:24.715174 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:24.715132 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:25:24.715544 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:24.715199 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" podUID="3a79e336-f4c4-4d28-8023-be246046f3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 00:25:25.715092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:25.715059 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:25:25.715267 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:25.715109 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" podUID="3a79e336-f4c4-4d28-8023-be246046f3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 00:25:26.715213 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:26.715182 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:25:26.715371 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:26.715243 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" podUID="3a79e336-f4c4-4d28-8023-be246046f3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 00:25:27.714873 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:27.714837 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:25:27.715046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:27.714921 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" podUID="3a79e336-f4c4-4d28-8023-be246046f3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 00:25:28.660605 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:25:28.660566 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:25:28.715333 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:28.715300 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:25:28.715488 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:28.715363 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" podUID="3a79e336-f4c4-4d28-8023-be246046f3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 00:25:29.694757 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:29.694670 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:25:29.695201 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:29.694822 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:25:29.719401 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:29.719379 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:29.719630 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:29.719615 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:29.720274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:29.720258 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:25:40.573800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.569942 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b"] Nov 03 00:25:40.593802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.588979 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:40.795800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.795194 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-73821c-openshift-default-dockercfg-cbk4d\"" Nov 03 00:25:40.901891 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.887795 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfww\" (UniqueName: \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-kube-api-access-dlfww\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:40.901891 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.887858 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-token\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:40.901891 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.887912 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-credential-socket\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:40.901891 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.887959 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-certs\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:40.901891 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.887987 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-socket\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:40.901891 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.888014 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-data\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:40.901891 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.888073 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-envoy\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:40.901891 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.888196 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istiod-ca-cert\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:40.901891 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:40.888310 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-podinfo\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.006848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.001030 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istiod-ca-cert\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.006848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.001156 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-podinfo\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.006848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.001232 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfww\" (UniqueName: \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-kube-api-access-dlfww\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.006848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.001267 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-token\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.006848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.001317 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-credential-socket\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.006848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.001366 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-certs\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.006848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.001392 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-socket\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.006848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.001421 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-data\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.006848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.001480 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-envoy\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.074806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.060136 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-data\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.116800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.106839 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" podUID="49aef8b0-ab85-4ccc-bb3d-c5eebfa27469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:41.116800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.108616 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istiod-ca-cert\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.116800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.116089 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-credential-socket\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.116800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.116365 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-certs\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.157882 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.120054 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-socket\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.188539 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.162473 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b"] Nov 03 00:25:41.202804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.190500 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-envoy\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.226396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.226361 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-token\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.240846 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.228429 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfww\" (UniqueName: \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-kube-api-access-dlfww\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.283928 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.283890 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-podinfo\") pod \"e2e-api-a-73821c-openshift-default-6888fd7597-td86b\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.301802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.298097 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:41.527855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.506097 4304 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" podUID="49aef8b0-ab85-4ccc-bb3d-c5eebfa27469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:41.537854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:41.536980 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" podUID="2fb1752b-050f-4042-8650-0be7a24c004e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:42.073880 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:42.066880 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" podUID="26d57814-f91a-423c-ab38-5cb55e4d77a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:42.237843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:42.126828 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" podUID="49aef8b0-ab85-4ccc-bb3d-c5eebfa27469" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:42.237843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:42.127699 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:25:42.237843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:42.155632 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6" Nov 03 00:25:42.237843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:42.203693 4304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" podUID="26d57814-f91a-423c-ab38-5cb55e4d77a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 03 00:25:42.237843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:42.224202 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" podUID="082c2451-3bb7-4b3a-82c0-f8ea634adcdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:42.525522 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:42.523467 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:42.525522 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:42.523553 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:42.525522 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:42.523589 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:42.579765 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:42.579719 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b"] Nov 03 00:25:43.331646 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:43.331599 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" event={"ID":"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1","Type":"ContainerStarted","Data":"4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4"} Nov 03 00:25:43.332160 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:43.331691 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" event={"ID":"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1","Type":"ContainerStarted","Data":"886e6cd01d60d11d5f83d9f5951fe8cc7b8bfa682362780ee978d0392025b9bf"} Nov 03 00:25:43.449450 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:43.449397 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" podStartSLOduration=3.449384427 podStartE2EDuration="3.449384427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:43.447592714 +0000 UTC m=+1226.731150388" watchObservedRunningTime="2026-11-03 00:25:43.449384427 +0000 UTC m=+1226.732942101" Nov 03 00:25:44.299419 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:44.299385 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:44.306046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:44.306018 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:44.337765 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:44.337735 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:44.338592 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:44.338574 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:25:47.574893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.574857 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w"] Nov 03 00:25:47.579044 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.579020 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:47.633099 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.633064 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cb65f3a5-serving-cert\"" Nov 03 00:25:47.673184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.673148 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3e2a93-5326-4d2d-bfc7-77f87727a863-openshift-service-ca\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:47.673335 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.673289 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-metrics-certs\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:47.673378 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.673335 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-maas-api-tls\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:47.673447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.673429 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h65f\" (UniqueName: \"kubernetes.io/projected/5b3e2a93-5326-4d2d-bfc7-77f87727a863-kube-api-access-8h65f\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:47.686071 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.686044 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w"] Nov 03 00:25:47.774963 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.774935 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-metrics-certs\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:47.775121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.774985 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-maas-api-tls\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:47.775121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.775048 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h65f\" (UniqueName: \"kubernetes.io/projected/5b3e2a93-5326-4d2d-bfc7-77f87727a863-kube-api-access-8h65f\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:47.775121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.775116 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3e2a93-5326-4d2d-bfc7-77f87727a863-openshift-service-ca\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:47.775224 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:25:47.775150 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-serving-cert: secret "maas-api-e2e-mt-cb65f3a5-serving-cert" not found Nov 03 00:25:47.775269 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:25:47.775223 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-maas-api-tls podName:5b3e2a93-5326-4d2d-bfc7-77f87727a863 nodeName:}" failed. No retries permitted until 2026-11-03 00:25:48.275206165 +0000 UTC m=+1231.558763820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-maas-api-tls") pod "maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" (UID: "5b3e2a93-5326-4d2d-bfc7-77f87727a863") : secret "maas-api-e2e-mt-cb65f3a5-serving-cert" not found Nov 03 00:25:47.775790 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.775761 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3e2a93-5326-4d2d-bfc7-77f87727a863-openshift-service-ca\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:47.777478 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.777460 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-metrics-certs\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:47.833473 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:47.833399 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h65f\" (UniqueName: \"kubernetes.io/projected/5b3e2a93-5326-4d2d-bfc7-77f87727a863-kube-api-access-8h65f\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:48.281583 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:48.281546 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-maas-api-tls\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:48.284168 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:48.284142 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-maas-api-tls\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:48.494566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:48.494533 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:48.876810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:48.876756 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w"] Nov 03 00:25:48.883855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:48.883830 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:25:49.369785 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:49.369737 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" event={"ID":"5b3e2a93-5326-4d2d-bfc7-77f87727a863","Type":"ContainerStarted","Data":"772420bc0edc9b01289f30ffe813676e8536bc051768a9e36efc031bbf95a9f6"} Nov 03 00:25:50.359788 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:50.359737 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk"] Nov 03 00:25:50.378019 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:50.377980 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" event={"ID":"5b3e2a93-5326-4d2d-bfc7-77f87727a863","Type":"ContainerStarted","Data":"041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3"} Nov 03 00:25:50.378167 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:50.378071 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:50.591635 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:50.591581 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" podStartSLOduration=2.665520365 podStartE2EDuration="3.59156507s" totalImagesPullingTime="926.044705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:50.589741246 +0000 UTC m=+1233.873298922" watchObservedRunningTime="2026-11-03 00:25:50.59156507 +0000 UTC m=+1233.875122791" Nov 03 00:25:55.361132 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.361073 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" podUID="93a05409-b3ab-457e-9458-3e8da111e6fa" containerName="payload-processing" containerID="cri-o://4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b" gracePeriod=25 Nov 03 00:25:55.635731 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.635705 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:25:55.784589 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.784553 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-plugins-config-volume\") pod \"93a05409-b3ab-457e-9458-3e8da111e6fa\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " Nov 03 00:25:55.784832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.784670 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-provider-credentials\") pod \"93a05409-b3ab-457e-9458-3e8da111e6fa\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " Nov 03 00:25:55.784832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.784733 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-routing-overlay-volume\") pod \"93a05409-b3ab-457e-9458-3e8da111e6fa\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " Nov 03 00:25:55.784978 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.784890 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-kube-api-access-2m5qd\" (UniqueName: \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-kube-api-access-2m5qd\") pod \"93a05409-b3ab-457e-9458-3e8da111e6fa\" (UID: \"93a05409-b3ab-457e-9458-3e8da111e6fa\") " Nov 03 00:25:55.785305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.785277 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-plugins-config-volume" pod "93a05409-b3ab-457e-9458-3e8da111e6fa" (UID: "93a05409-b3ab-457e-9458-3e8da111e6fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:55.785598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.785566 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-routing-overlay-volume" pod "93a05409-b3ab-457e-9458-3e8da111e6fa" (UID: "93a05409-b3ab-457e-9458-3e8da111e6fa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:55.787490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.787466 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-provider-credentials" pod "93a05409-b3ab-457e-9458-3e8da111e6fa" (UID: "93a05409-b3ab-457e-9458-3e8da111e6fa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:55.787604 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.787577 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-kube-api-access-2m5qd" pod "93a05409-b3ab-457e-9458-3e8da111e6fa" (UID: "93a05409-b3ab-457e-9458-3e8da111e6fa"). InnerVolumeSpecName "kube-api-access-2m5qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:55.886933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.886859 4304 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-provider-credentials\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:55.886933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.886885 4304 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-routing-overlay-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:55.886933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.886896 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m5qd\" (UniqueName: \"kubernetes.io/projected/93a05409-b3ab-457e-9458-3e8da111e6fa-kube-api-access-2m5qd\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:55.886933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:55.886905 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a05409-b3ab-457e-9458-3e8da111e6fa-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:56.396701 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:56.396670 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:25:56.425028 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:56.424997 4304 generic.go:356] "Generic (PLEG): container finished" podID="93a05409-b3ab-457e-9458-3e8da111e6fa" containerID="4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b" exitCode=0 Nov 03 00:25:56.425201 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:56.425071 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" Nov 03 00:25:56.425201 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:56.425072 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" event={"ID":"93a05409-b3ab-457e-9458-3e8da111e6fa","Type":"ContainerDied","Data":"4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b"} Nov 03 00:25:56.425327 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:56.425205 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk" event={"ID":"93a05409-b3ab-457e-9458-3e8da111e6fa","Type":"ContainerDied","Data":"48492ef4fa6dd86afaa8e0a2b5e9c7bd1e008ea2d4bb596404c9163c1f7f9b8d"} Nov 03 00:25:56.425327 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:56.425237 4304 scope.go:122] "RemoveContainer" containerID="4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b" Nov 03 00:25:56.435206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:56.435188 4304 scope.go:122] "RemoveContainer" containerID="4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b" Nov 03 00:25:56.435435 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:25:56.435414 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b\": container with ID starting with 4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b not found: ID does not exist" containerID="4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b" Nov 03 00:25:56.435490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:56.435445 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b"} err="failed to get container status \"4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b\": rpc error: code = NotFound desc = could not find container \"4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b\": container with ID starting with 4b33f72479daacc14fbc854ac5ef27f39ddcec5be731a9063a1f731de515a33b not found: ID does not exist" Nov 03 00:25:56.623349 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:56.623315 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk"] Nov 03 00:25:56.626435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:56.626414 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85c6f868cb-krhkk"] Nov 03 00:25:56.829901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:56.829810 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a05409-b3ab-457e-9458-3e8da111e6fa" path="/var/lib/kubelet/pods/93a05409-b3ab-457e-9458-3e8da111e6fa/volumes" Nov 03 00:25:58.606262 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.606232 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx"] Nov 03 00:25:58.606803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.606788 4304 cpu_manager.go:414] "containerMap: removing container" podUID="93a05409-b3ab-457e-9458-3e8da111e6fa" containerName="payload-processing" Nov 03 00:25:58.606851 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.606804 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a05409-b3ab-457e-9458-3e8da111e6fa" containerName="payload-processing" Nov 03 00:25:58.606941 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.606931 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a05409-b3ab-457e-9458-3e8da111e6fa" containerName="payload-processing" Nov 03 00:25:58.610350 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.610333 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.616912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.616881 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/135bebde-c159-440a-8b6c-2ae327db3d7e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.617032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.616934 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.617153 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.617133 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.617254 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.617234 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.617309 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.617289 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.617412 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.617381 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.617750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.617488 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxjx\" (UniqueName: \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-kube-api-access-jjxjx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.656170 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:25:58.656128 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:25:58.662023 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.662001 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 00:25:58.662144 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.662108 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 00:25:58.662144 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.662124 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 00:25:58.711545 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.711505 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gnssc\"" Nov 03 00:25:58.715461 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.715434 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx"] Nov 03 00:25:58.718660 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.718631 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.718839 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.718702 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.718839 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.718729 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.718988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.718820 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.719103 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.719083 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxjx\" (UniqueName: \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-kube-api-access-jjxjx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.719188 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.719172 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/135bebde-c159-440a-8b6c-2ae327db3d7e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.719246 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.719208 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.719690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.719659 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.722097 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.722048 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.722444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.722420 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.723026 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.723005 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/135bebde-c159-440a-8b6c-2ae327db3d7e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.723214 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.723192 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.776260 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.776228 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.776547 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.776522 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxjx\" (UniqueName: \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-kube-api-access-jjxjx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26xpx\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:58.922413 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:58.922384 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:25:59.348255 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:59.348222 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx"] Nov 03 00:25:59.348601 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:25:59.348571 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135bebde_c159_440a_8b6c_2ae327db3d7e.slice/crio-cf9a563383fb0a3cb2808d75a15bee525253bbe9e1b95641b6d5c344fb057a6a WatchSource:0}: Error finding container cf9a563383fb0a3cb2808d75a15bee525253bbe9e1b95641b6d5c344fb057a6a: Status 404 returned error can't find the container with id cf9a563383fb0a3cb2808d75a15bee525253bbe9e1b95641b6d5c344fb057a6a Nov 03 00:25:59.451054 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:25:59.451019 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" event={"ID":"135bebde-c159-440a-8b6c-2ae327db3d7e","Type":"ContainerStarted","Data":"cf9a563383fb0a3cb2808d75a15bee525253bbe9e1b95641b6d5c344fb057a6a"} Nov 03 00:26:04.708800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:04.706952 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:26:04.708800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:04.707021 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:26:05.888944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:05.888897 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:26:05.889470 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:05.888978 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:26:06.104333 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:06.104289 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:26:06.104333 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:06.104377 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:26:06.593002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:06.592915 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" event={"ID":"135bebde-c159-440a-8b6c-2ae327db3d7e","Type":"ContainerStarted","Data":"8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50"} Nov 03 00:26:06.593002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:06.592986 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:26:06.810835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:06.810767 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" podStartSLOduration=2.059657944 podStartE2EDuration="8.810752347s" totalImagesPullingTime="6.751094403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:06.757977292 +0000 UTC m=+1250.041534978" watchObservedRunningTime="2026-11-03 00:26:06.810752347 +0000 UTC m=+1250.094310020" Nov 03 00:26:07.600833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:07.600799 4304 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-26xpx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:26:07.601335 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:07.600848 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" podUID="135bebde-c159-440a-8b6c-2ae327db3d7e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:26:10.599917 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:26:10.599888 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:26:28.657202 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:26:28.657162 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:26:58.656606 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:26:58.656565 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:27:00.379988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:00.379907 4304 patch_prober.go:29] interesting pod/e2e-api-a-73821c-openshift-default-6888fd7597-td86b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:27:00.380349 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:00.379978 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:27:01.010089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:01.010012 4304 generic.go:356] "Generic (PLEG): container finished" podID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerID="4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4" exitCode=0 Nov 03 00:27:01.010274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:01.010084 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" event={"ID":"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1","Type":"ContainerDied","Data":"4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4"} Nov 03 00:27:01.010524 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:01.010511 4304 scope.go:122] "RemoveContainer" containerID="4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4" Nov 03 00:27:01.012099 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:01.012064 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:01.012193 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:01.012164 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:01.012244 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:01.012227 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:02.018380 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:02.018347 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" event={"ID":"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1","Type":"ContainerStarted","Data":"4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38"} Nov 03 00:27:02.299625 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:02.299539 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:27:02.299824 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:02.299710 4304 patch_prober.go:29] interesting pod/e2e-api-a-73821c-openshift-default-6888fd7597-td86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:27:02.299824 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:02.299762 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 00:27:03.299619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:03.299580 4304 patch_prober.go:29] interesting pod/e2e-api-a-73821c-openshift-default-6888fd7597-td86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:27:03.300078 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:03.299648 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 00:27:04.299811 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:04.299747 4304 patch_prober.go:29] interesting pod/e2e-api-a-73821c-openshift-default-6888fd7597-td86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:27:04.300238 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:04.299840 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 00:27:05.299583 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:05.299550 4304 patch_prober.go:29] interesting pod/e2e-api-a-73821c-openshift-default-6888fd7597-td86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:27:05.299762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:05.299614 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 00:27:06.299339 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:06.299306 4304 patch_prober.go:29] interesting pod/e2e-api-a-73821c-openshift-default-6888fd7597-td86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:27:06.299741 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:06.299378 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 00:27:07.302539 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:07.302499 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:27:07.302916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:07.302802 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:27:07.303593 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:07.303576 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:27:23.326109 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.324796 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6"] Nov 03 00:27:23.334875 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.334749 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:27:23.369856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.369826 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6"] Nov 03 00:27:23.386191 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.386159 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjkjp\" (UniqueName: \"kubernetes.io/projected/b9315860-a8bf-4d57-b752-2170be50dd40-kube-api-access-vjkjp\") pod \"payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6\" (UID: \"b9315860-a8bf-4d57-b752-2170be50dd40\") " pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:27:23.386348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.386312 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9315860-a8bf-4d57-b752-2170be50dd40-plugins-config-volume\") pod \"payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6\" (UID: \"b9315860-a8bf-4d57-b752-2170be50dd40\") " pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:27:23.437243 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.437219 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-be578d-dockercfg-cmqz9\"" Nov 03 00:27:23.437243 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.437219 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-be578d\"" Nov 03 00:27:23.487820 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.487762 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjkjp\" (UniqueName: \"kubernetes.io/projected/b9315860-a8bf-4d57-b752-2170be50dd40-kube-api-access-vjkjp\") pod \"payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6\" (UID: \"b9315860-a8bf-4d57-b752-2170be50dd40\") " pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:27:23.487994 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.487972 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9315860-a8bf-4d57-b752-2170be50dd40-plugins-config-volume\") pod \"payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6\" (UID: \"b9315860-a8bf-4d57-b752-2170be50dd40\") " pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:27:23.488684 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.488657 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9315860-a8bf-4d57-b752-2170be50dd40-plugins-config-volume\") pod \"payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6\" (UID: \"b9315860-a8bf-4d57-b752-2170be50dd40\") " pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:27:23.544233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.544201 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjkjp\" (UniqueName: \"kubernetes.io/projected/b9315860-a8bf-4d57-b752-2170be50dd40-kube-api-access-vjkjp\") pod \"payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6\" (UID: \"b9315860-a8bf-4d57-b752-2170be50dd40\") " pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:27:23.672720 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:23.672686 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:27:24.048655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:24.048627 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6"] Nov 03 00:27:24.049048 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:27:24.049025 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9315860_a8bf_4d57_b752_2170be50dd40.slice/crio-c507d011f435c46297788e085cf803138834a89ab302134d3c5c4b2b84cc6444 WatchSource:0}: Error finding container c507d011f435c46297788e085cf803138834a89ab302134d3c5c4b2b84cc6444: Status 404 returned error can't find the container with id c507d011f435c46297788e085cf803138834a89ab302134d3c5c4b2b84cc6444 Nov 03 00:27:24.375665 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:24.375549 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" event={"ID":"b9315860-a8bf-4d57-b752-2170be50dd40","Type":"ContainerStarted","Data":"c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e"} Nov 03 00:27:24.376166 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:24.375668 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" event={"ID":"b9315860-a8bf-4d57-b752-2170be50dd40","Type":"ContainerStarted","Data":"c507d011f435c46297788e085cf803138834a89ab302134d3c5c4b2b84cc6444"} Nov 03 00:27:24.376166 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:24.375893 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:27:24.490876 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:24.490804 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" podStartSLOduration=1.490769118 podStartE2EDuration="1.490769118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:24.488575678 +0000 UTC m=+1327.772133353" watchObservedRunningTime="2026-11-03 00:27:24.490769118 +0000 UTC m=+1327.774326794" Nov 03 00:27:28.088156 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.088125 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c"] Nov 03 00:27:28.092136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.092112 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:27:28.150832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.150803 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c"] Nov 03 00:27:28.156115 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.156078 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beb868e6-f7a8-4f84-84f4-ed55953decb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c\" (UID: \"beb868e6-f7a8-4f84-84f4-ed55953decb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:27:28.156342 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.156319 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxlf\" (UniqueName: \"kubernetes.io/projected/beb868e6-f7a8-4f84-84f4-ed55953decb6-kube-api-access-rfxlf\") pod \"payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c\" (UID: \"beb868e6-f7a8-4f84-84f4-ed55953decb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:27:28.256830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.256395 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-fd6ea3\"" Nov 03 00:27:28.256830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.256503 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-fd6ea3-dockercfg-f495d\"" Nov 03 00:27:28.296833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.295566 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxlf\" (UniqueName: \"kubernetes.io/projected/beb868e6-f7a8-4f84-84f4-ed55953decb6-kube-api-access-rfxlf\") pod \"payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c\" (UID: \"beb868e6-f7a8-4f84-84f4-ed55953decb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:27:28.296833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.295700 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beb868e6-f7a8-4f84-84f4-ed55953decb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c\" (UID: \"beb868e6-f7a8-4f84-84f4-ed55953decb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:27:28.296833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.296596 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beb868e6-f7a8-4f84-84f4-ed55953decb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c\" (UID: \"beb868e6-f7a8-4f84-84f4-ed55953decb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:27:28.352509 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.352433 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxlf\" (UniqueName: \"kubernetes.io/projected/beb868e6-f7a8-4f84-84f4-ed55953decb6-kube-api-access-rfxlf\") pod \"payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c\" (UID: \"beb868e6-f7a8-4f84-84f4-ed55953decb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:27:28.406339 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.406312 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:27:28.655262 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:27:28.655228 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:27:28.791083 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:28.787503 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c"] Nov 03 00:27:28.792923 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:27:28.792894 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb868e6_f7a8_4f84_84f4_ed55953decb6.slice/crio-98043a86ed7ae6f6507e15f8f7572e0b4254735c3119ebc146c795d81a3f975c WatchSource:0}: Error finding container 98043a86ed7ae6f6507e15f8f7572e0b4254735c3119ebc146c795d81a3f975c: Status 404 returned error can't find the container with id 98043a86ed7ae6f6507e15f8f7572e0b4254735c3119ebc146c795d81a3f975c Nov 03 00:27:29.408749 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:29.408712 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" event={"ID":"beb868e6-f7a8-4f84-84f4-ed55953decb6","Type":"ContainerStarted","Data":"89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742"} Nov 03 00:27:29.409193 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:29.408798 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" event={"ID":"beb868e6-f7a8-4f84-84f4-ed55953decb6","Type":"ContainerStarted","Data":"98043a86ed7ae6f6507e15f8f7572e0b4254735c3119ebc146c795d81a3f975c"} Nov 03 00:27:29.409193 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:29.408825 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:27:29.521993 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:29.521934 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" podStartSLOduration=1.5219147020000001 podStartE2EDuration="1.521914702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:29.521495065 +0000 UTC m=+1332.805052739" watchObservedRunningTime="2026-11-03 00:27:29.521914702 +0000 UTC m=+1332.805472377" Nov 03 00:27:35.385237 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:35.385201 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:27:40.418308 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:27:40.418274 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:27:58.655088 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:27:58.655051 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:28:28.656591 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:28:28.656553 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:28:58.657018 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:28:58.656939 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:29:18.017168 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.017128 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf"] Nov 03 00:29:18.021264 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.021247 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.073596 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.073549 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c521ba-openshift-default-dockercfg-sp9kk\"" Nov 03 00:29:18.197847 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.173850 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-288s2\" (UniqueName: \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-kube-api-access-288s2\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.227798 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.198210 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-credential-socket\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.228096 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.228065 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-envoy\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.236639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.228282 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-certs\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.236639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.236100 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-data\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.236639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.236172 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-istio-token\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.236639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.236243 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833edfe8-d264-4331-947f-6648bd594ff8-istio-podinfo\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.236639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.236317 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-socket\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.236639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.236344 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833edfe8-d264-4331-947f-6648bd594ff8-istiod-ca-cert\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.252610 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.251369 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf"] Nov 03 00:29:18.337447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.337336 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-288s2\" (UniqueName: \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-kube-api-access-288s2\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.337447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.337412 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-credential-socket\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.337447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.337436 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-envoy\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.337712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.337486 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-certs\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.337712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.337514 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-data\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.337712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.337539 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-istio-token\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.337712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.337597 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833edfe8-d264-4331-947f-6648bd594ff8-istio-podinfo\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.337712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.337642 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-socket\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.337712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.337663 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833edfe8-d264-4331-947f-6648bd594ff8-istiod-ca-cert\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.338035 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.338016 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-data\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.338226 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.338210 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-credential-socket\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.338445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.338421 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833edfe8-d264-4331-947f-6648bd594ff8-istiod-ca-cert\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.338538 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.338471 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-certs\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.338538 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.338513 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-socket\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.341189 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.341167 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833edfe8-d264-4331-947f-6648bd594ff8-istio-podinfo\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.341304 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.341222 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-envoy\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.395332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.395304 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-288s2\" (UniqueName: \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-kube-api-access-288s2\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.395519 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.395369 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-istio-token\") pod \"e2e-api-b-c521ba-openshift-default-646df59755-75nrf\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:18.636093 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:18.635911 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:19.086118 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:29:19.086088 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833edfe8_d264_4331_947f_6648bd594ff8.slice/crio-1f79faeb902da9f6f0c71e922d1c0f341cfc76c55168cdab57c87570a05dc6d6 WatchSource:0}: Error finding container 1f79faeb902da9f6f0c71e922d1c0f341cfc76c55168cdab57c87570a05dc6d6: Status 404 returned error can't find the container with id 1f79faeb902da9f6f0c71e922d1c0f341cfc76c55168cdab57c87570a05dc6d6 Nov 03 00:29:19.086468 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:19.086344 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf"] Nov 03 00:29:19.088742 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:19.088707 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:29:19.088879 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:19.088811 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:29:19.088879 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:19.088844 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:29:19.290729 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:19.290697 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" event={"ID":"833edfe8-d264-4331-947f-6648bd594ff8","Type":"ContainerStarted","Data":"9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36"} Nov 03 00:29:19.290960 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:19.290757 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" event={"ID":"833edfe8-d264-4331-947f-6648bd594ff8","Type":"ContainerStarted","Data":"1f79faeb902da9f6f0c71e922d1c0f341cfc76c55168cdab57c87570a05dc6d6"} Nov 03 00:29:19.407532 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:19.407468 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" podStartSLOduration=2.4074519580000002 podStartE2EDuration="2.407451958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:19.406225286 +0000 UTC m=+1442.689782961" watchObservedRunningTime="2026-11-03 00:29:19.407451958 +0000 UTC m=+1442.691009633" Nov 03 00:29:19.637041 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:19.637013 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:20.641634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:20.641606 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:21.303528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:21.303498 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:21.304503 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:21.304484 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:29:25.862229 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:25.862196 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8"] Nov 03 00:29:25.968689 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:25.968657 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n"] Nov 03 00:29:28.655626 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:29:28.655591 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:29:30.444972 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.389925 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz"] Nov 03 00:29:30.460565 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.460520 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:29:30.470034 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.470002 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz"] Nov 03 00:29:30.512414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.512393 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-38f5a3\"" Nov 03 00:29:30.512583 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.512492 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-38f5a3-dockercfg-fhdw8\"" Nov 03 00:29:30.573908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.573877 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvd85\" (UniqueName: \"kubernetes.io/projected/5be96e00-c9a0-4c5b-9fa7-34af12668de0-kube-api-access-zvd85\") pod \"payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz\" (UID: \"5be96e00-c9a0-4c5b-9fa7-34af12668de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:29:30.574107 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.574086 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be96e00-c9a0-4c5b-9fa7-34af12668de0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz\" (UID: \"5be96e00-c9a0-4c5b-9fa7-34af12668de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:29:30.675422 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.675388 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be96e00-c9a0-4c5b-9fa7-34af12668de0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz\" (UID: \"5be96e00-c9a0-4c5b-9fa7-34af12668de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:29:30.675589 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.675552 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvd85\" (UniqueName: \"kubernetes.io/projected/5be96e00-c9a0-4c5b-9fa7-34af12668de0-kube-api-access-zvd85\") pod \"payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz\" (UID: \"5be96e00-c9a0-4c5b-9fa7-34af12668de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:29:30.676168 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.676149 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be96e00-c9a0-4c5b-9fa7-34af12668de0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz\" (UID: \"5be96e00-c9a0-4c5b-9fa7-34af12668de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:29:30.732320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.732255 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvd85\" (UniqueName: \"kubernetes.io/projected/5be96e00-c9a0-4c5b-9fa7-34af12668de0-kube-api-access-zvd85\") pod \"payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz\" (UID: \"5be96e00-c9a0-4c5b-9fa7-34af12668de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:29:30.791820 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.791761 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:29:30.863216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.863008 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" podUID="ea296694-ea86-47cb-8e07-9470e09e1755" containerName="payload-processing" containerID="cri-o://18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17" gracePeriod=25 Nov 03 00:29:30.969502 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:30.969457 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" podUID="c69b9a91-3d04-430c-9ce1-58444ca13c00" containerName="payload-pre-processing" containerID="cri-o://627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb" gracePeriod=25 Nov 03 00:29:31.177894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:31.177863 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz"] Nov 03 00:29:31.178843 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:29:31.178820 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be96e00_c9a0_4c5b_9fa7_34af12668de0.slice/crio-bd24e86711740f5f68c2dd0764886860789b7fb61ad7474ed6d6ca5034476581 WatchSource:0}: Error finding container bd24e86711740f5f68c2dd0764886860789b7fb61ad7474ed6d6ca5034476581: Status 404 returned error can't find the container with id bd24e86711740f5f68c2dd0764886860789b7fb61ad7474ed6d6ca5034476581 Nov 03 00:29:31.483694 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:31.483606 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" event={"ID":"5be96e00-c9a0-4c5b-9fa7-34af12668de0","Type":"ContainerStarted","Data":"cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd"} Nov 03 00:29:31.483694 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:31.483674 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" event={"ID":"5be96e00-c9a0-4c5b-9fa7-34af12668de0","Type":"ContainerStarted","Data":"bd24e86711740f5f68c2dd0764886860789b7fb61ad7474ed6d6ca5034476581"} Nov 03 00:29:31.484135 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:31.483700 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:29:31.600200 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:31.600145 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" podStartSLOduration=1.600131617 podStartE2EDuration="1.600131617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:31.597911874 +0000 UTC m=+1454.881469571" watchObservedRunningTime="2026-11-03 00:29:31.600131617 +0000 UTC m=+1454.883689291" Nov 03 00:29:32.993746 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:32.976473 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt"] Nov 03 00:29:33.042767 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.042563 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt"] Nov 03 00:29:33.043384 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.043074 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:29:33.114719 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.114691 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589736ae-24c5-4a6a-8adc-039de3029fe0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c521ba-86875995bf-9mltt\" (UID: \"589736ae-24c5-4a6a-8adc-039de3029fe0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:29:33.114974 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.114959 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tctm\" (UniqueName: \"kubernetes.io/projected/589736ae-24c5-4a6a-8adc-039de3029fe0-kube-api-access-5tctm\") pod \"payload-processing-e2e-api-b-c521ba-86875995bf-9mltt\" (UID: \"589736ae-24c5-4a6a-8adc-039de3029fe0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:29:33.145043 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.145016 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c521ba\"" Nov 03 00:29:33.145043 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.145047 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c521ba-dockercfg-c96s8\"" Nov 03 00:29:33.215619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.215587 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tctm\" (UniqueName: \"kubernetes.io/projected/589736ae-24c5-4a6a-8adc-039de3029fe0-kube-api-access-5tctm\") pod \"payload-processing-e2e-api-b-c521ba-86875995bf-9mltt\" (UID: \"589736ae-24c5-4a6a-8adc-039de3029fe0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:29:33.215619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.215626 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589736ae-24c5-4a6a-8adc-039de3029fe0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c521ba-86875995bf-9mltt\" (UID: \"589736ae-24c5-4a6a-8adc-039de3029fe0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:29:33.216512 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.216486 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589736ae-24c5-4a6a-8adc-039de3029fe0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c521ba-86875995bf-9mltt\" (UID: \"589736ae-24c5-4a6a-8adc-039de3029fe0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:29:33.273260 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.273179 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tctm\" (UniqueName: \"kubernetes.io/projected/589736ae-24c5-4a6a-8adc-039de3029fe0-kube-api-access-5tctm\") pod \"payload-processing-e2e-api-b-c521ba-86875995bf-9mltt\" (UID: \"589736ae-24c5-4a6a-8adc-039de3029fe0\") " pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:29:33.367470 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.367416 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:29:33.743691 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:29:33.743661 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589736ae_24c5_4a6a_8adc_039de3029fe0.slice/crio-5bb49c3ff32e98814a2b46a679b1854d4e33d53b76447f563bbcb4ae64e055bd WatchSource:0}: Error finding container 5bb49c3ff32e98814a2b46a679b1854d4e33d53b76447f563bbcb4ae64e055bd: Status 404 returned error can't find the container with id 5bb49c3ff32e98814a2b46a679b1854d4e33d53b76447f563bbcb4ae64e055bd Nov 03 00:29:33.744068 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:33.744043 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt"] Nov 03 00:29:34.564638 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:34.564590 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" event={"ID":"589736ae-24c5-4a6a-8adc-039de3029fe0","Type":"ContainerStarted","Data":"e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91"} Nov 03 00:29:34.565098 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:34.564699 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:29:34.565098 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:34.564724 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" event={"ID":"589736ae-24c5-4a6a-8adc-039de3029fe0","Type":"ContainerStarted","Data":"5bb49c3ff32e98814a2b46a679b1854d4e33d53b76447f563bbcb4ae64e055bd"} Nov 03 00:29:34.726433 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:34.726362 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" podStartSLOduration=2.726343589 podStartE2EDuration="2.726343589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:34.676821688 +0000 UTC m=+1457.960379363" watchObservedRunningTime="2026-11-03 00:29:34.726343589 +0000 UTC m=+1458.009901264" Nov 03 00:29:39.692919 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:39.692879 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" podUID="c69b9a91-3d04-430c-9ce1-58444ca13c00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:29:39.693315 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:39.692886 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" podUID="ea296694-ea86-47cb-8e07-9470e09e1755" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:29:42.551193 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:42.551163 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:29:45.575055 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:45.575027 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:29:49.692882 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:49.692845 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" podUID="c69b9a91-3d04-430c-9ce1-58444ca13c00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:29:49.693324 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:49.692856 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" podUID="ea296694-ea86-47cb-8e07-9470e09e1755" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:29:51.150493 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.150463 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:29:51.216007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.215392 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea296694-ea86-47cb-8e07-9470e09e1755-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea296694-ea86-47cb-8e07-9470e09e1755-plugins-config-volume\") pod \"ea296694-ea86-47cb-8e07-9470e09e1755\" (UID: \"ea296694-ea86-47cb-8e07-9470e09e1755\") " Nov 03 00:29:51.216007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.215452 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea296694-ea86-47cb-8e07-9470e09e1755-kube-api-access-zfjkj\" (UniqueName: \"kubernetes.io/projected/ea296694-ea86-47cb-8e07-9470e09e1755-kube-api-access-zfjkj\") pod \"ea296694-ea86-47cb-8e07-9470e09e1755\" (UID: \"ea296694-ea86-47cb-8e07-9470e09e1755\") " Nov 03 00:29:51.216007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.215818 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea296694-ea86-47cb-8e07-9470e09e1755-plugins-config-volume" pod "ea296694-ea86-47cb-8e07-9470e09e1755" (UID: "ea296694-ea86-47cb-8e07-9470e09e1755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:51.216821 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.216761 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea296694-ea86-47cb-8e07-9470e09e1755-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:51.218148 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.218116 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea296694-ea86-47cb-8e07-9470e09e1755-kube-api-access-zfjkj" pod "ea296694-ea86-47cb-8e07-9470e09e1755" (UID: "ea296694-ea86-47cb-8e07-9470e09e1755"). InnerVolumeSpecName "kube-api-access-zfjkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:51.245571 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.245550 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:29:51.318038 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.318005 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69b9a91-3d04-430c-9ce1-58444ca13c00-kube-api-access-g2s8f\" (UniqueName: \"kubernetes.io/projected/c69b9a91-3d04-430c-9ce1-58444ca13c00-kube-api-access-g2s8f\") pod \"c69b9a91-3d04-430c-9ce1-58444ca13c00\" (UID: \"c69b9a91-3d04-430c-9ce1-58444ca13c00\") " Nov 03 00:29:51.318241 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.318229 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c69b9a91-3d04-430c-9ce1-58444ca13c00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69b9a91-3d04-430c-9ce1-58444ca13c00-plugins-config-volume\") pod \"c69b9a91-3d04-430c-9ce1-58444ca13c00\" (UID: \"c69b9a91-3d04-430c-9ce1-58444ca13c00\") " Nov 03 00:29:51.318636 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.318609 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69b9a91-3d04-430c-9ce1-58444ca13c00-plugins-config-volume" pod "c69b9a91-3d04-430c-9ce1-58444ca13c00" (UID: "c69b9a91-3d04-430c-9ce1-58444ca13c00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:51.318759 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.318703 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfjkj\" (UniqueName: \"kubernetes.io/projected/ea296694-ea86-47cb-8e07-9470e09e1755-kube-api-access-zfjkj\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:51.320278 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.320255 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69b9a91-3d04-430c-9ce1-58444ca13c00-kube-api-access-g2s8f" pod "c69b9a91-3d04-430c-9ce1-58444ca13c00" (UID: "c69b9a91-3d04-430c-9ce1-58444ca13c00"). InnerVolumeSpecName "kube-api-access-g2s8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:51.425422 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.425382 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69b9a91-3d04-430c-9ce1-58444ca13c00-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:51.425422 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.425423 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2s8f\" (UniqueName: \"kubernetes.io/projected/c69b9a91-3d04-430c-9ce1-58444ca13c00-kube-api-access-g2s8f\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:51.683944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.683864 4304 generic.go:356] "Generic (PLEG): container finished" podID="c69b9a91-3d04-430c-9ce1-58444ca13c00" containerID="627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb" exitCode=0 Nov 03 00:29:51.683944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.683914 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" Nov 03 00:29:51.684145 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.683951 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" event={"ID":"c69b9a91-3d04-430c-9ce1-58444ca13c00","Type":"ContainerDied","Data":"627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb"} Nov 03 00:29:51.684145 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.684002 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n" event={"ID":"c69b9a91-3d04-430c-9ce1-58444ca13c00","Type":"ContainerDied","Data":"fc18c74dc4aaefa16f22b3ccadf5188a776d5ff8b3024240ac38956de189ca97"} Nov 03 00:29:51.684145 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.684032 4304 scope.go:122] "RemoveContainer" containerID="627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb" Nov 03 00:29:51.685716 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.685689 4304 generic.go:356] "Generic (PLEG): container finished" podID="ea296694-ea86-47cb-8e07-9470e09e1755" containerID="18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17" exitCode=0 Nov 03 00:29:51.685868 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.685801 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" event={"ID":"ea296694-ea86-47cb-8e07-9470e09e1755","Type":"ContainerDied","Data":"18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17"} Nov 03 00:29:51.685868 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.685831 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" Nov 03 00:29:51.685868 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.685856 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8" event={"ID":"ea296694-ea86-47cb-8e07-9470e09e1755","Type":"ContainerDied","Data":"927413eacab0a5611b3c9743e3402305fa86039febe72a00ac264b4bd2b9e58e"} Nov 03 00:29:51.699474 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.698863 4304 scope.go:122] "RemoveContainer" containerID="627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb" Nov 03 00:29:51.699474 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:29:51.699402 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb\": container with ID starting with 627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb not found: ID does not exist" containerID="627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb" Nov 03 00:29:51.699474 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.699447 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb"} err="failed to get container status \"627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb\": rpc error: code = NotFound desc = could not find container \"627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb\": container with ID starting with 627e3f525b9fe746ccc5d7e273b0b2e77da8facf9500d1c6975abedebff54dfb not found: ID does not exist" Nov 03 00:29:51.699474 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.699471 4304 scope.go:122] "RemoveContainer" containerID="18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17" Nov 03 00:29:51.709842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.709825 4304 scope.go:122] "RemoveContainer" containerID="18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17" Nov 03 00:29:51.710124 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:29:51.710104 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17\": container with ID starting with 18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17 not found: ID does not exist" containerID="18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17" Nov 03 00:29:51.710162 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.710132 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17"} err="failed to get container status \"18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17\": rpc error: code = NotFound desc = could not find container \"18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17\": container with ID starting with 18c4c568155655d900249d715cc5ceae22d842b5d40acb4e6d51b5867951de17 not found: ID does not exist" Nov 03 00:29:51.856459 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.856409 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8"] Nov 03 00:29:51.905965 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.905937 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-gg2h8"] Nov 03 00:29:51.970429 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.970331 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n"] Nov 03 00:29:51.973718 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:51.973697 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-msg4n"] Nov 03 00:29:52.830291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:52.830257 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69b9a91-3d04-430c-9ce1-58444ca13c00" path="/var/lib/kubelet/pods/c69b9a91-3d04-430c-9ce1-58444ca13c00/volumes" Nov 03 00:29:52.830868 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:52.830850 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea296694-ea86-47cb-8e07-9470e09e1755" path="/var/lib/kubelet/pods/ea296694-ea86-47cb-8e07-9470e09e1755/volumes" Nov 03 00:29:58.654800 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:29:58.654739 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:29:58.800258 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:58.800225 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b"] Nov 03 00:29:58.800474 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:29:58.800440 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" containerID="cri-o://4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38" gracePeriod=30 Nov 03 00:30:00.207348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.207269 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj"] Nov 03 00:30:00.207901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.207886 4304 cpu_manager.go:414] "containerMap: removing container" podUID="c69b9a91-3d04-430c-9ce1-58444ca13c00" containerName="payload-pre-processing" Nov 03 00:30:00.207971 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.207903 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69b9a91-3d04-430c-9ce1-58444ca13c00" containerName="payload-pre-processing" Nov 03 00:30:00.207971 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.207949 4304 cpu_manager.go:414] "containerMap: removing container" podUID="ea296694-ea86-47cb-8e07-9470e09e1755" containerName="payload-processing" Nov 03 00:30:00.207971 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.207955 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea296694-ea86-47cb-8e07-9470e09e1755" containerName="payload-processing" Nov 03 00:30:00.208066 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.208041 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea296694-ea86-47cb-8e07-9470e09e1755" containerName="payload-processing" Nov 03 00:30:00.208066 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.208050 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69b9a91-3d04-430c-9ce1-58444ca13c00" containerName="payload-pre-processing" Nov 03 00:30:00.213508 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.213489 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" Nov 03 00:30:00.320573 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.320535 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj"] Nov 03 00:30:00.357598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.357561 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6326520-8f7d-425a-86e4-ba63ece9e79a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj\" (UID: \"f6326520-8f7d-425a-86e4-ba63ece9e79a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" Nov 03 00:30:00.357598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.357605 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph5cb\" (UniqueName: \"kubernetes.io/projected/f6326520-8f7d-425a-86e4-ba63ece9e79a-kube-api-access-ph5cb\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj\" (UID: \"f6326520-8f7d-425a-86e4-ba63ece9e79a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" Nov 03 00:30:00.365522 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.365491 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:30:00.459485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.459393 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6326520-8f7d-425a-86e4-ba63ece9e79a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj\" (UID: \"f6326520-8f7d-425a-86e4-ba63ece9e79a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" Nov 03 00:30:00.459485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.459440 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph5cb\" (UniqueName: \"kubernetes.io/projected/f6326520-8f7d-425a-86e4-ba63ece9e79a-kube-api-access-ph5cb\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj\" (UID: \"f6326520-8f7d-425a-86e4-ba63ece9e79a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" Nov 03 00:30:00.460058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.460039 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6326520-8f7d-425a-86e4-ba63ece9e79a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj\" (UID: \"f6326520-8f7d-425a-86e4-ba63ece9e79a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" Nov 03 00:30:00.515677 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.515643 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph5cb\" (UniqueName: \"kubernetes.io/projected/f6326520-8f7d-425a-86e4-ba63ece9e79a-kube-api-access-ph5cb\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj\" (UID: \"f6326520-8f7d-425a-86e4-ba63ece9e79a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" Nov 03 00:30:00.527560 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.527534 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" Nov 03 00:30:00.909468 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:00.909439 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj"] Nov 03 00:30:00.909940 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:30:00.909883 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6326520_8f7d_425a_86e4_ba63ece9e79a.slice/crio-40e95e156691642962b43b88242bbb7d42c5b10df3d51e38569db0f4373af266 WatchSource:0}: Error finding container 40e95e156691642962b43b88242bbb7d42c5b10df3d51e38569db0f4373af266: Status 404 returned error can't find the container with id 40e95e156691642962b43b88242bbb7d42c5b10df3d51e38569db0f4373af266 Nov 03 00:30:01.768555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:01.768517 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" event={"ID":"f6326520-8f7d-425a-86e4-ba63ece9e79a","Type":"ContainerStarted","Data":"40e95e156691642962b43b88242bbb7d42c5b10df3d51e38569db0f4373af266"} Nov 03 00:30:04.092294 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.092265 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:30:04.212800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.212740 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istiod-ca-cert\") pod \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " Nov 03 00:30:04.212954 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.212838 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-envoy\") pod \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " Nov 03 00:30:04.213049 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213030 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-podinfo\") pod \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " Nov 03 00:30:04.213096 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213070 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-certs\") pod \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " Nov 03 00:30:04.213139 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213103 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-data\") pod \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " Nov 03 00:30:04.213139 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213114 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istiod-ca-cert" pod "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" (UID: "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:04.213257 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213238 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-socket\") pod \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " Nov 03 00:30:04.213340 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213324 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-kube-api-access-dlfww\" (UniqueName: \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-kube-api-access-dlfww\") pod \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " Nov 03 00:30:04.213469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213359 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-credential-socket\") pod \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " Nov 03 00:30:04.213469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213363 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-certs" pod "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" (UID: "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:04.213469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213404 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-token\" (UniqueName: \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-token\") pod \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\" (UID: \"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1\") " Nov 03 00:30:04.213469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213461 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-data" pod "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" (UID: "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:04.213682 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213546 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-socket" pod "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" (UID: "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:04.213801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.213738 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-credential-socket" pod "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" (UID: "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:04.214212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.214188 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:04.214212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.214215 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:04.214536 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.214231 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:04.214536 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.214245 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:04.214536 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.214258 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:04.215334 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.215312 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-podinfo" pod "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" (UID: "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:30:04.215424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.215389 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-envoy" pod "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" (UID: "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:04.215510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.215491 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-kube-api-access-dlfww" pod "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" (UID: "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1"). InnerVolumeSpecName "kube-api-access-dlfww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:04.215973 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.215959 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-token" pod "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" (UID: "9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:04.315638 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.315565 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:04.315638 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.315592 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlfww\" (UniqueName: \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-kube-api-access-dlfww\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:04.315638 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.315604 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:04.315638 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.315612 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:04.789351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.789315 4304 generic.go:356] "Generic (PLEG): container finished" podID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerID="4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38" exitCode=0 Nov 03 00:30:04.789548 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.789376 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" Nov 03 00:30:04.789548 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.789404 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" event={"ID":"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1","Type":"ContainerDied","Data":"4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38"} Nov 03 00:30:04.789548 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.789463 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b" event={"ID":"9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1","Type":"ContainerDied","Data":"886e6cd01d60d11d5f83d9f5951fe8cc7b8bfa682362780ee978d0392025b9bf"} Nov 03 00:30:04.789548 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.789491 4304 scope.go:122] "RemoveContainer" containerID="4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38" Nov 03 00:30:04.800676 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.800661 4304 scope.go:122] "RemoveContainer" containerID="4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4" Nov 03 00:30:04.809930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.809911 4304 scope.go:122] "RemoveContainer" containerID="4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38" Nov 03 00:30:04.810183 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:30:04.810164 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38\": container with ID starting with 4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38 not found: ID does not exist" containerID="4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38" Nov 03 00:30:04.810254 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.810197 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38"} err="failed to get container status \"4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38\": rpc error: code = NotFound desc = could not find container \"4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38\": container with ID starting with 4a68a3065d01cb5a8af5110a9c6f4212230cf81928882a8783c753718dacfb38 not found: ID does not exist" Nov 03 00:30:04.810254 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.810223 4304 scope.go:122] "RemoveContainer" containerID="4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4" Nov 03 00:30:04.810493 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:30:04.810475 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4\": container with ID starting with 4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4 not found: ID does not exist" containerID="4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4" Nov 03 00:30:04.810537 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.810498 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4"} err="failed to get container status \"4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4\": rpc error: code = NotFound desc = could not find container \"4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4\": container with ID starting with 4ea7e2bf0aef809f51c3d5b7ba27bacba32ff741c03299ea660d6e33359baaa4 not found: ID does not exist" Nov 03 00:30:04.954961 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.954932 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b"] Nov 03 00:30:04.958417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:04.958391 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-73821c-openshift-default-6888fd7597-td86b"] Nov 03 00:30:06.120243 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:06.120210 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c"] Nov 03 00:30:06.830342 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:06.830308 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" path="/var/lib/kubelet/pods/9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1/volumes" Nov 03 00:30:11.120840 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.120814 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" podUID="beb868e6-f7a8-4f84-84f4-ed55953decb6" containerName="payload-pre-processing" containerID="cri-o://89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742" gracePeriod=25 Nov 03 00:30:11.223421 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.223392 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8"] Nov 03 00:30:11.223625 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.223592 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerName="istio-proxy" containerID="cri-o://a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198" gracePeriod=30 Nov 03 00:30:11.451427 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.451399 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:30:11.502698 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.502670 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb868e6-f7a8-4f84-84f4-ed55953decb6-kube-api-access-rfxlf\" (UniqueName: \"kubernetes.io/projected/beb868e6-f7a8-4f84-84f4-ed55953decb6-kube-api-access-rfxlf\") pod \"beb868e6-f7a8-4f84-84f4-ed55953decb6\" (UID: \"beb868e6-f7a8-4f84-84f4-ed55953decb6\") " Nov 03 00:30:11.502914 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.502798 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb868e6-f7a8-4f84-84f4-ed55953decb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beb868e6-f7a8-4f84-84f4-ed55953decb6-plugins-config-volume\") pod \"beb868e6-f7a8-4f84-84f4-ed55953decb6\" (UID: \"beb868e6-f7a8-4f84-84f4-ed55953decb6\") " Nov 03 00:30:11.503169 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.503136 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb868e6-f7a8-4f84-84f4-ed55953decb6-plugins-config-volume" pod "beb868e6-f7a8-4f84-84f4-ed55953decb6" (UID: "beb868e6-f7a8-4f84-84f4-ed55953decb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:11.503329 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.503312 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beb868e6-f7a8-4f84-84f4-ed55953decb6-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:11.505022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.504999 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb868e6-f7a8-4f84-84f4-ed55953decb6-kube-api-access-rfxlf" pod "beb868e6-f7a8-4f84-84f4-ed55953decb6" (UID: "beb868e6-f7a8-4f84-84f4-ed55953decb6"). InnerVolumeSpecName "kube-api-access-rfxlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:11.605176 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.605133 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfxlf\" (UniqueName: \"kubernetes.io/projected/beb868e6-f7a8-4f84-84f4-ed55953decb6-kube-api-access-rfxlf\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:11.849370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.849284 4304 generic.go:356] "Generic (PLEG): container finished" podID="beb868e6-f7a8-4f84-84f4-ed55953decb6" containerID="89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742" exitCode=0 Nov 03 00:30:11.849370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.849331 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" Nov 03 00:30:11.849593 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.849365 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" event={"ID":"beb868e6-f7a8-4f84-84f4-ed55953decb6","Type":"ContainerDied","Data":"89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742"} Nov 03 00:30:11.849593 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.849412 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c" event={"ID":"beb868e6-f7a8-4f84-84f4-ed55953decb6","Type":"ContainerDied","Data":"98043a86ed7ae6f6507e15f8f7572e0b4254735c3119ebc146c795d81a3f975c"} Nov 03 00:30:11.849593 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.849433 4304 scope.go:122] "RemoveContainer" containerID="89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742" Nov 03 00:30:11.851207 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.851169 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" event={"ID":"f6326520-8f7d-425a-86e4-ba63ece9e79a","Type":"ContainerStarted","Data":"3724b42bc4f15bc253b1e1b7646da93dafb1295d09883b3867623a9314ef919b"} Nov 03 00:30:11.860225 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.860211 4304 scope.go:122] "RemoveContainer" containerID="89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742" Nov 03 00:30:11.860480 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:30:11.860463 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742\": container with ID starting with 89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742 not found: ID does not exist" containerID="89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742" Nov 03 00:30:11.860521 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:11.860487 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742"} err="failed to get container status \"89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742\": rpc error: code = NotFound desc = could not find container \"89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742\": container with ID starting with 89e47ea3a1d284de546d7a889fb2647e5a6e6f7255bc1c233ef9e7d3b5754742 not found: ID does not exist" Nov 03 00:30:12.013579 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:12.013547 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c"] Nov 03 00:30:12.016600 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:12.016579 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd6ea3-79888c5d99-7f22c"] Nov 03 00:30:12.131197 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:12.131107 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" podStartSLOduration=1.926942676 podStartE2EDuration="12.131092823s" totalImagesPullingTime="10.204150147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:12.129342059 +0000 UTC m=+1495.412899733" watchObservedRunningTime="2026-11-03 00:30:12.131092823 +0000 UTC m=+1495.414650497" Nov 03 00:30:12.829375 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:12.829343 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb868e6-f7a8-4f84-84f4-ed55953decb6" path="/var/lib/kubelet/pods/beb868e6-f7a8-4f84-84f4-ed55953decb6/volumes" Nov 03 00:30:12.859235 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:12.859205 4304 generic.go:356] "Generic (PLEG): container finished" podID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerID="3724b42bc4f15bc253b1e1b7646da93dafb1295d09883b3867623a9314ef919b" exitCode=60 Nov 03 00:30:12.859399 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:12.859302 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" event={"ID":"f6326520-8f7d-425a-86e4-ba63ece9e79a","Type":"ContainerDied","Data":"3724b42bc4f15bc253b1e1b7646da93dafb1295d09883b3867623a9314ef919b"} Nov 03 00:30:12.859552 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:12.859538 4304 scope.go:122] "RemoveContainer" containerID="3724b42bc4f15bc253b1e1b7646da93dafb1295d09883b3867623a9314ef919b" Nov 03 00:30:13.867052 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:13.867021 4304 generic.go:356] "Generic (PLEG): container finished" podID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerID="d0700fb64b894edc42135c6857837aa045927b1390184cd3260bc0e1a5da9a9a" exitCode=60 Nov 03 00:30:13.867440 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:13.867106 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" event={"ID":"f6326520-8f7d-425a-86e4-ba63ece9e79a","Type":"ContainerDied","Data":"d0700fb64b894edc42135c6857837aa045927b1390184cd3260bc0e1a5da9a9a"} Nov 03 00:30:13.867440 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:13.867190 4304 scope.go:122] "RemoveContainer" containerID="3724b42bc4f15bc253b1e1b7646da93dafb1295d09883b3867623a9314ef919b" Nov 03 00:30:13.867440 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:13.867342 4304 scope.go:122] "RemoveContainer" containerID="d0700fb64b894edc42135c6857837aa045927b1390184cd3260bc0e1a5da9a9a" Nov 03 00:30:13.867630 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:30:13.867609 4304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj_odh-ai-gateway-infra(f6326520-8f7d-425a-86e4-ba63ece9e79a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" Nov 03 00:30:14.875662 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:14.875633 4304 scope.go:122] "RemoveContainer" containerID="d0700fb64b894edc42135c6857837aa045927b1390184cd3260bc0e1a5da9a9a" Nov 03 00:30:14.876066 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:30:14.875915 4304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj_odh-ai-gateway-infra(f6326520-8f7d-425a-86e4-ba63ece9e79a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" Nov 03 00:30:16.517789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.517741 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:30:16.572562 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.572530 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-token\" (UniqueName: \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-token\") pod \"05faa1fc-ab4e-49ba-935c-27de58df56f1\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " Nov 03 00:30:16.572742 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.572569 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-podinfo\") pod \"05faa1fc-ab4e-49ba-935c-27de58df56f1\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " Nov 03 00:30:16.572742 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.572603 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-kube-api-access-qnvg5\" (UniqueName: \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-kube-api-access-qnvg5\") pod \"05faa1fc-ab4e-49ba-935c-27de58df56f1\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " Nov 03 00:30:16.572742 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.572727 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-socket\") pod \"05faa1fc-ab4e-49ba-935c-27de58df56f1\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " Nov 03 00:30:16.572968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.572763 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05faa1fc-ab4e-49ba-935c-27de58df56f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05faa1fc-ab4e-49ba-935c-27de58df56f1-istiod-ca-cert\") pod \"05faa1fc-ab4e-49ba-935c-27de58df56f1\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " Nov 03 00:30:16.572968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.572921 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-data\") pod \"05faa1fc-ab4e-49ba-935c-27de58df56f1\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " Nov 03 00:30:16.573084 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.572984 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-certs\") pod \"05faa1fc-ab4e-49ba-935c-27de58df56f1\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " Nov 03 00:30:16.573084 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.573008 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-socket" pod "05faa1fc-ab4e-49ba-935c-27de58df56f1" (UID: "05faa1fc-ab4e-49ba-935c-27de58df56f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:16.573182 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.573096 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-envoy\") pod \"05faa1fc-ab4e-49ba-935c-27de58df56f1\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " Nov 03 00:30:16.573182 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.573150 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-credential-socket\") pod \"05faa1fc-ab4e-49ba-935c-27de58df56f1\" (UID: \"05faa1fc-ab4e-49ba-935c-27de58df56f1\") " Nov 03 00:30:16.573292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.573174 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-data" pod "05faa1fc-ab4e-49ba-935c-27de58df56f1" (UID: "05faa1fc-ab4e-49ba-935c-27de58df56f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:16.573292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.573227 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05faa1fc-ab4e-49ba-935c-27de58df56f1-istiod-ca-cert" pod "05faa1fc-ab4e-49ba-935c-27de58df56f1" (UID: "05faa1fc-ab4e-49ba-935c-27de58df56f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:16.573292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.573278 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-certs" pod "05faa1fc-ab4e-49ba-935c-27de58df56f1" (UID: "05faa1fc-ab4e-49ba-935c-27de58df56f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:16.573601 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.573576 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-credential-socket" pod "05faa1fc-ab4e-49ba-935c-27de58df56f1" (UID: "05faa1fc-ab4e-49ba-935c-27de58df56f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:16.574170 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.574143 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.574284 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.574173 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.574284 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.574191 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05faa1fc-ab4e-49ba-935c-27de58df56f1-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.574284 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.574206 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.574284 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.574220 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.574994 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.574967 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-podinfo" pod "05faa1fc-ab4e-49ba-935c-27de58df56f1" (UID: "05faa1fc-ab4e-49ba-935c-27de58df56f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:30:16.575122 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.575080 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-token" pod "05faa1fc-ab4e-49ba-935c-27de58df56f1" (UID: "05faa1fc-ab4e-49ba-935c-27de58df56f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:16.575214 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.575195 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-kube-api-access-qnvg5" pod "05faa1fc-ab4e-49ba-935c-27de58df56f1" (UID: "05faa1fc-ab4e-49ba-935c-27de58df56f1"). InnerVolumeSpecName "kube-api-access-qnvg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:16.575368 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.575352 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-envoy" pod "05faa1fc-ab4e-49ba-935c-27de58df56f1" (UID: "05faa1fc-ab4e-49ba-935c-27de58df56f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:16.675283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.675250 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.675283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.675278 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.675485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.675294 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05faa1fc-ab4e-49ba-935c-27de58df56f1-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.675485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.675309 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnvg5\" (UniqueName: \"kubernetes.io/projected/05faa1fc-ab4e-49ba-935c-27de58df56f1-kube-api-access-qnvg5\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.889608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.889575 4304 generic.go:356] "Generic (PLEG): container finished" podID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerID="a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198" exitCode=0 Nov 03 00:30:16.889608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.889603 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" event={"ID":"05faa1fc-ab4e-49ba-935c-27de58df56f1","Type":"ContainerDied","Data":"a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198"} Nov 03 00:30:16.889877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.889625 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" Nov 03 00:30:16.889877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.889660 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8" event={"ID":"05faa1fc-ab4e-49ba-935c-27de58df56f1","Type":"ContainerDied","Data":"968dfe7387119cb71a8ce8967233416e5d4a924ac318602796c871b0617f86e0"} Nov 03 00:30:16.889877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.889685 4304 scope.go:122] "RemoveContainer" containerID="a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198" Nov 03 00:30:16.899500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.899485 4304 scope.go:122] "RemoveContainer" containerID="a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198" Nov 03 00:30:16.899768 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:30:16.899749 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198\": container with ID starting with a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198 not found: ID does not exist" containerID="a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198" Nov 03 00:30:16.899871 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:16.899802 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198"} err="failed to get container status \"a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198\": rpc error: code = NotFound desc = could not find container \"a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198\": container with ID starting with a7a446b126a10774a60ac3d3bd9b5d5c2e5a2c8a4cef898380898dee9711e198 not found: ID does not exist" Nov 03 00:30:17.054754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:17.054724 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8"] Nov 03 00:30:17.073971 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:17.073932 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-zg8r8"] Nov 03 00:30:18.831323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:18.831286 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" path="/var/lib/kubelet/pods/05faa1fc-ab4e-49ba-935c-27de58df56f1/volumes" Nov 03 00:30:21.774646 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.774608 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9"] Nov 03 00:30:21.775566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.775532 4304 cpu_manager.go:414] "containerMap: removing container" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerName="istio-proxy" Nov 03 00:30:21.775566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.775566 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerName="istio-proxy" Nov 03 00:30:21.775762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.775579 4304 cpu_manager.go:414] "containerMap: removing container" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" Nov 03 00:30:21.775762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.775584 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" Nov 03 00:30:21.775762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.775615 4304 cpu_manager.go:414] "containerMap: removing container" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" Nov 03 00:30:21.775762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.775621 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" Nov 03 00:30:21.775762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.775631 4304 cpu_manager.go:414] "containerMap: removing container" podUID="beb868e6-f7a8-4f84-84f4-ed55953decb6" containerName="payload-pre-processing" Nov 03 00:30:21.775762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.775635 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb868e6-f7a8-4f84-84f4-ed55953decb6" containerName="payload-pre-processing" Nov 03 00:30:21.775762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.775769 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb868e6-f7a8-4f84-84f4-ed55953decb6" containerName="payload-pre-processing" Nov 03 00:30:21.776141 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.775806 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="05faa1fc-ab4e-49ba-935c-27de58df56f1" containerName="istio-proxy" Nov 03 00:30:21.776141 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.775815 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" Nov 03 00:30:21.780897 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.780878 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.851797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.850689 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-token\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.851797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.850814 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-data\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.855797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.852332 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d8924d-ef4a-4891-9492-b257cbdc7647-istiod-ca-cert\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.855797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.852404 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-envoy\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.855797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.852431 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkfjj\" (UniqueName: \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-kube-api-access-lkfjj\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.855797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.852483 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-podinfo\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.855797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.852581 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-socket\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.855797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.852611 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-credential-socket\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.855797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.852651 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-certs\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.904349 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.899022 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bb5643-openshift-default-dockercfg-d89ct\"" Nov 03 00:30:21.910054 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.909493 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9"] Nov 03 00:30:21.954356 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.954303 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-certs\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.954592 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.954398 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-token\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.954592 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.954490 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-data\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.954592 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.954580 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d8924d-ef4a-4891-9492-b257cbdc7647-istiod-ca-cert\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.954855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.954634 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-envoy\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.954855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.954660 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkfjj\" (UniqueName: \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-kube-api-access-lkfjj\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.954855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.954713 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-podinfo\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.954855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.954833 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-socket\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.955084 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.954864 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-credential-socket\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.955354 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.955329 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-credential-socket\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.957133 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.956058 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-certs\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.957292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.956190 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d8924d-ef4a-4891-9492-b257cbdc7647-istiod-ca-cert\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.957411 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.956534 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-data\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.957525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.957065 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-socket\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.962083 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.962062 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-podinfo\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:21.962557 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:21.962540 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-envoy\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:22.014917 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:22.014881 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-token\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:22.016323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:22.016304 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkfjj\" (UniqueName: \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-kube-api-access-lkfjj\") pod \"e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:22.112992 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:22.112908 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:22.545952 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:22.545922 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9"] Nov 03 00:30:22.547568 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:30:22.547545 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d8924d_ef4a_4891_9492_b257cbdc7647.slice/crio-6a95e58b92461284530ffc399a37a1b960707c2bc831a5ac50617bb1928ea14e WatchSource:0}: Error finding container 6a95e58b92461284530ffc399a37a1b960707c2bc831a5ac50617bb1928ea14e: Status 404 returned error can't find the container with id 6a95e58b92461284530ffc399a37a1b960707c2bc831a5ac50617bb1928ea14e Nov 03 00:30:22.550217 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:22.550184 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:30:22.550309 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:22.550248 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:30:22.550309 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:22.550288 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:30:22.954620 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:22.954585 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" event={"ID":"c2d8924d-ef4a-4891-9492-b257cbdc7647","Type":"ContainerStarted","Data":"05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759"} Nov 03 00:30:22.955026 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:22.954637 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" event={"ID":"c2d8924d-ef4a-4891-9492-b257cbdc7647","Type":"ContainerStarted","Data":"6a95e58b92461284530ffc399a37a1b960707c2bc831a5ac50617bb1928ea14e"} Nov 03 00:30:23.074281 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:23.074234 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" podStartSLOduration=2.074220245 podStartE2EDuration="2.074220245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:23.070247367 +0000 UTC m=+1506.353805041" watchObservedRunningTime="2026-11-03 00:30:23.074220245 +0000 UTC m=+1506.357777918" Nov 03 00:30:23.113875 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:23.113844 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:23.119726 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:23.119701 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:23.705176 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:23.705143 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r"] Nov 03 00:30:23.705359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:23.705342 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" podUID="3a79e336-f4c4-4d28-8023-be246046f3dc" containerName="istio-proxy" containerID="cri-o://ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84" gracePeriod=30 Nov 03 00:30:23.961752 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:23.961669 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:23.962686 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:23.962661 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:30:27.822834 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:27.822800 4304 scope.go:122] "RemoveContainer" containerID="d0700fb64b894edc42135c6857837aa045927b1390184cd3260bc0e1a5da9a9a" Nov 03 00:30:28.655801 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:30:28.655746 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:30:28.979650 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:28.979627 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:30:28.995100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:28.995073 4304 generic.go:356] "Generic (PLEG): container finished" podID="3a79e336-f4c4-4d28-8023-be246046f3dc" containerID="ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84" exitCode=0 Nov 03 00:30:28.995223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:28.995117 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" Nov 03 00:30:28.995223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:28.995155 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" event={"ID":"3a79e336-f4c4-4d28-8023-be246046f3dc","Type":"ContainerDied","Data":"ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84"} Nov 03 00:30:28.995316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:28.995244 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r" event={"ID":"3a79e336-f4c4-4d28-8023-be246046f3dc","Type":"ContainerDied","Data":"0e36c998f7d521455a222dbd43ac27cf8838b939ccf7d0750cd338ebe79c9fed"} Nov 03 00:30:28.995316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:28.995272 4304 scope.go:122] "RemoveContainer" containerID="ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84" Nov 03 00:30:28.997065 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:28.997045 4304 generic.go:356] "Generic (PLEG): container finished" podID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerID="7d11609a5fbd8d8b3e7f2a63057200ff75eea8f0900027980e2688dac80bc693" exitCode=60 Nov 03 00:30:28.997170 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:28.997072 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" event={"ID":"f6326520-8f7d-425a-86e4-ba63ece9e79a","Type":"ContainerDied","Data":"7d11609a5fbd8d8b3e7f2a63057200ff75eea8f0900027980e2688dac80bc693"} Nov 03 00:30:28.997421 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:28.997409 4304 scope.go:122] "RemoveContainer" containerID="7d11609a5fbd8d8b3e7f2a63057200ff75eea8f0900027980e2688dac80bc693" Nov 03 00:30:28.997703 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:30:28.997682 4304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj_odh-ai-gateway-infra(f6326520-8f7d-425a-86e4-ba63ece9e79a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" Nov 03 00:30:29.004867 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.004850 4304 scope.go:122] "RemoveContainer" containerID="ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84" Nov 03 00:30:29.005107 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:30:29.005090 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84\": container with ID starting with ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84 not found: ID does not exist" containerID="ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84" Nov 03 00:30:29.005149 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.005116 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84"} err="failed to get container status \"ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84\": rpc error: code = NotFound desc = could not find container \"ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84\": container with ID starting with ebec21564606fd77369a7a64dfb262639e1d05b297189945c97ab6e54a75bd84 not found: ID does not exist" Nov 03 00:30:29.005149 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.005132 4304 scope.go:122] "RemoveContainer" containerID="d0700fb64b894edc42135c6857837aa045927b1390184cd3260bc0e1a5da9a9a" Nov 03 00:30:29.157690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.157661 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-podinfo\") pod \"3a79e336-f4c4-4d28-8023-be246046f3dc\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " Nov 03 00:30:29.157907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.157873 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-envoy\") pod \"3a79e336-f4c4-4d28-8023-be246046f3dc\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " Nov 03 00:30:29.157978 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.157942 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-credential-socket\") pod \"3a79e336-f4c4-4d28-8023-be246046f3dc\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " Nov 03 00:30:29.157978 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.157972 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-token\" (UniqueName: \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-token\") pod \"3a79e336-f4c4-4d28-8023-be246046f3dc\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " Nov 03 00:30:29.158096 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.158022 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a79e336-f4c4-4d28-8023-be246046f3dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a79e336-f4c4-4d28-8023-be246046f3dc-istiod-ca-cert\") pod \"3a79e336-f4c4-4d28-8023-be246046f3dc\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " Nov 03 00:30:29.158096 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.158052 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-socket\") pod \"3a79e336-f4c4-4d28-8023-be246046f3dc\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " Nov 03 00:30:29.158096 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.158082 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-data\") pod \"3a79e336-f4c4-4d28-8023-be246046f3dc\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " Nov 03 00:30:29.158296 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.158134 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-kube-api-access-rvqp2\" (UniqueName: \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-kube-api-access-rvqp2\") pod \"3a79e336-f4c4-4d28-8023-be246046f3dc\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " Nov 03 00:30:29.158296 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.158182 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-certs\") pod \"3a79e336-f4c4-4d28-8023-be246046f3dc\" (UID: \"3a79e336-f4c4-4d28-8023-be246046f3dc\") " Nov 03 00:30:29.158748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.158610 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-socket" pod "3a79e336-f4c4-4d28-8023-be246046f3dc" (UID: "3a79e336-f4c4-4d28-8023-be246046f3dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:29.158866 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.158836 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a79e336-f4c4-4d28-8023-be246046f3dc-istiod-ca-cert" pod "3a79e336-f4c4-4d28-8023-be246046f3dc" (UID: "3a79e336-f4c4-4d28-8023-be246046f3dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:29.158939 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.158861 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.159500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.159048 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-credential-socket" pod "3a79e336-f4c4-4d28-8023-be246046f3dc" (UID: "3a79e336-f4c4-4d28-8023-be246046f3dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:29.159500 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.159455 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-data" pod "3a79e336-f4c4-4d28-8023-be246046f3dc" (UID: "3a79e336-f4c4-4d28-8023-be246046f3dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:29.159880 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.159847 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-certs" pod "3a79e336-f4c4-4d28-8023-be246046f3dc" (UID: "3a79e336-f4c4-4d28-8023-be246046f3dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:29.161004 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.160980 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-envoy" pod "3a79e336-f4c4-4d28-8023-be246046f3dc" (UID: "3a79e336-f4c4-4d28-8023-be246046f3dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:29.162406 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.162382 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-podinfo" pod "3a79e336-f4c4-4d28-8023-be246046f3dc" (UID: "3a79e336-f4c4-4d28-8023-be246046f3dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:30:29.162915 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.162874 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-kube-api-access-rvqp2" pod "3a79e336-f4c4-4d28-8023-be246046f3dc" (UID: "3a79e336-f4c4-4d28-8023-be246046f3dc"). InnerVolumeSpecName "kube-api-access-rvqp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:29.163068 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.163046 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-token" pod "3a79e336-f4c4-4d28-8023-be246046f3dc" (UID: "3a79e336-f4c4-4d28-8023-be246046f3dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:29.260518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.260444 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.260518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.260471 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.260518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.260482 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.260518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.260491 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.260518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.260499 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a79e336-f4c4-4d28-8023-be246046f3dc-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.260518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.260507 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.260518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.260516 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvqp2\" (UniqueName: \"kubernetes.io/projected/3a79e336-f4c4-4d28-8023-be246046f3dc-kube-api-access-rvqp2\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.260885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.260527 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a79e336-f4c4-4d28-8023-be246046f3dc-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.436468 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.436428 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r"] Nov 03 00:30:29.442734 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:29.442701 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-sb99r"] Nov 03 00:30:30.247732 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:30.247696 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj"] Nov 03 00:30:30.410714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:30.410690 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" Nov 03 00:30:30.585642 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:30.585557 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6326520-8f7d-425a-86e4-ba63ece9e79a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6326520-8f7d-425a-86e4-ba63ece9e79a-maas-api-service-ca\") pod \"f6326520-8f7d-425a-86e4-ba63ece9e79a\" (UID: \"f6326520-8f7d-425a-86e4-ba63ece9e79a\") " Nov 03 00:30:30.585861 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:30.585719 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6326520-8f7d-425a-86e4-ba63ece9e79a-kube-api-access-ph5cb\" (UniqueName: \"kubernetes.io/projected/f6326520-8f7d-425a-86e4-ba63ece9e79a-kube-api-access-ph5cb\") pod \"f6326520-8f7d-425a-86e4-ba63ece9e79a\" (UID: \"f6326520-8f7d-425a-86e4-ba63ece9e79a\") " Nov 03 00:30:30.586005 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:30.585981 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6326520-8f7d-425a-86e4-ba63ece9e79a-maas-api-service-ca" pod "f6326520-8f7d-425a-86e4-ba63ece9e79a" (UID: "f6326520-8f7d-425a-86e4-ba63ece9e79a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:30.586369 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:30.586317 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6326520-8f7d-425a-86e4-ba63ece9e79a-maas-api-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:30.587895 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:30.587870 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6326520-8f7d-425a-86e4-ba63ece9e79a-kube-api-access-ph5cb" pod "f6326520-8f7d-425a-86e4-ba63ece9e79a" (UID: "f6326520-8f7d-425a-86e4-ba63ece9e79a"). InnerVolumeSpecName "kube-api-access-ph5cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:30.687221 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:30.687190 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph5cb\" (UniqueName: \"kubernetes.io/projected/f6326520-8f7d-425a-86e4-ba63ece9e79a-kube-api-access-ph5cb\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:30.829672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:30.829638 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a79e336-f4c4-4d28-8023-be246046f3dc" path="/var/lib/kubelet/pods/3a79e336-f4c4-4d28-8023-be246046f3dc/volumes" Nov 03 00:30:31.014338 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:31.014303 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" event={"ID":"f6326520-8f7d-425a-86e4-ba63ece9e79a","Type":"ContainerDied","Data":"40e95e156691642962b43b88242bbb7d42c5b10df3d51e38569db0f4373af266"} Nov 03 00:30:31.014498 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:31.014345 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj" Nov 03 00:30:31.014498 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:31.014372 4304 scope.go:122] "RemoveContainer" containerID="7d11609a5fbd8d8b3e7f2a63057200ff75eea8f0900027980e2688dac80bc693" Nov 03 00:30:31.177964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:31.177937 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj"] Nov 03 00:30:31.183596 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:31.183571 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894430-6cbvj"] Nov 03 00:30:32.829975 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:32.829935 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" path="/var/lib/kubelet/pods/f6326520-8f7d-425a-86e4-ba63ece9e79a/volumes" Nov 03 00:30:43.159713 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.159679 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w"] Nov 03 00:30:43.160264 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.159907 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" podUID="5b3e2a93-5326-4d2d-bfc7-77f87727a863" containerName="maas-api" containerID="cri-o://041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3" gracePeriod=30 Nov 03 00:30:43.556802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.556372 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:30:43.673061 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.660886 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3e2a93-5326-4d2d-bfc7-77f87727a863-kube-api-access-8h65f\" (UniqueName: \"kubernetes.io/projected/5b3e2a93-5326-4d2d-bfc7-77f87727a863-kube-api-access-8h65f\") pod \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " Nov 03 00:30:43.673061 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.660965 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3e2a93-5326-4d2d-bfc7-77f87727a863-kube-api-access-8h65f" pod "5b3e2a93-5326-4d2d-bfc7-77f87727a863" (UID: "5b3e2a93-5326-4d2d-bfc7-77f87727a863"). InnerVolumeSpecName "kube-api-access-8h65f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:43.673061 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.661122 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3e2a93-5326-4d2d-bfc7-77f87727a863-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3e2a93-5326-4d2d-bfc7-77f87727a863-openshift-service-ca\") pod \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " Nov 03 00:30:43.673061 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.661232 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-metrics-certs\") pod \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " Nov 03 00:30:43.673061 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.661265 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-maas-api-tls\") pod \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\" (UID: \"5b3e2a93-5326-4d2d-bfc7-77f87727a863\") " Nov 03 00:30:43.673061 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.661918 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h65f\" (UniqueName: \"kubernetes.io/projected/5b3e2a93-5326-4d2d-bfc7-77f87727a863-kube-api-access-8h65f\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:43.680988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.674985 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3e2a93-5326-4d2d-bfc7-77f87727a863-openshift-service-ca" pod "5b3e2a93-5326-4d2d-bfc7-77f87727a863" (UID: "5b3e2a93-5326-4d2d-bfc7-77f87727a863"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:43.695808 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.688651 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-maas-api-tls" pod "5b3e2a93-5326-4d2d-bfc7-77f87727a863" (UID: "5b3e2a93-5326-4d2d-bfc7-77f87727a863"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:30:43.695808 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.690856 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-metrics-certs" pod "5b3e2a93-5326-4d2d-bfc7-77f87727a863" (UID: "5b3e2a93-5326-4d2d-bfc7-77f87727a863"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:30:43.762877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.762756 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:43.762877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.762817 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3e2a93-5326-4d2d-bfc7-77f87727a863-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:43.762877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:43.762833 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3e2a93-5326-4d2d-bfc7-77f87727a863-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:44.212623 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:44.212586 4304 generic.go:356] "Generic (PLEG): container finished" podID="5b3e2a93-5326-4d2d-bfc7-77f87727a863" containerID="041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3" exitCode=0 Nov 03 00:30:44.213331 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:44.213307 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" event={"ID":"5b3e2a93-5326-4d2d-bfc7-77f87727a863","Type":"ContainerDied","Data":"041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3"} Nov 03 00:30:44.213547 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:44.213531 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" event={"ID":"5b3e2a93-5326-4d2d-bfc7-77f87727a863","Type":"ContainerDied","Data":"772420bc0edc9b01289f30ffe813676e8536bc051768a9e36efc031bbf95a9f6"} Nov 03 00:30:44.213665 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:44.213653 4304 scope.go:122] "RemoveContainer" containerID="041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3" Nov 03 00:30:44.213942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:44.213926 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w" Nov 03 00:30:44.262210 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:44.261427 4304 scope.go:122] "RemoveContainer" containerID="041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3" Nov 03 00:30:44.262614 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:30:44.262506 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3\": container with ID starting with 041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3 not found: ID does not exist" containerID="041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3" Nov 03 00:30:44.262614 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:44.262545 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3"} err="failed to get container status \"041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3\": rpc error: code = NotFound desc = could not find container \"041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3\": container with ID starting with 041faa208c5b53f0646d0a8122637f3769eb6ccfc63ff7d92e91849e1e601fd3 not found: ID does not exist" Nov 03 00:30:44.386473 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:44.386395 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w"] Nov 03 00:30:44.436033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:44.436002 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-hgv5w"] Nov 03 00:30:44.830005 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:44.829972 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3e2a93-5326-4d2d-bfc7-77f87727a863" path="/var/lib/kubelet/pods/5b3e2a93-5326-4d2d-bfc7-77f87727a863/volumes" Nov 03 00:30:48.246363 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:48.246330 4304 generic.go:356] "Generic (PLEG): container finished" podID="833edfe8-d264-4331-947f-6648bd594ff8" containerID="9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36" exitCode=0 Nov 03 00:30:48.246803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:48.246403 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" event={"ID":"833edfe8-d264-4331-947f-6648bd594ff8","Type":"ContainerDied","Data":"9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36"} Nov 03 00:30:48.246850 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:48.246835 4304 scope.go:122] "RemoveContainer" containerID="9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36" Nov 03 00:30:48.248447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:48.248424 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:30:48.248521 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:48.248508 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:30:48.248559 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:48.248543 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:30:49.255088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:49.255055 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" event={"ID":"833edfe8-d264-4331-947f-6648bd594ff8","Type":"ContainerStarted","Data":"eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108"} Nov 03 00:30:49.645868 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:49.641495 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:30:49.646197 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:49.646169 4304 patch_prober.go:29] interesting pod/e2e-api-b-c521ba-openshift-default-646df59755-75nrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:30:49.646398 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:49.646366 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 00:30:50.641800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:50.641053 4304 patch_prober.go:29] interesting pod/e2e-api-b-c521ba-openshift-default-646df59755-75nrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:30:50.641800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:50.641116 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 00:30:51.304311 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:51.304277 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:30:51.645862 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:51.644192 4304 patch_prober.go:29] interesting pod/e2e-api-b-c521ba-openshift-default-646df59755-75nrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:30:51.645862 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:51.644262 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 00:30:52.645930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:52.645896 4304 patch_prober.go:29] interesting pod/e2e-api-b-c521ba-openshift-default-646df59755-75nrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:30:52.646415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:52.645958 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 00:30:53.645973 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:53.645930 4304 patch_prober.go:29] interesting pod/e2e-api-b-c521ba-openshift-default-646df59755-75nrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:30:53.646963 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:53.646924 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 00:30:54.641380 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:54.641352 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:30:54.642245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:30:54.642228 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:30:58.655596 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:30:58.655552 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:31:04.710213 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:04.710179 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz"] Nov 03 00:31:09.711461 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:09.711407 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" podUID="5be96e00-c9a0-4c5b-9fa7-34af12668de0" containerName="payload-processing" containerID="cri-o://cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd" gracePeriod=25 Nov 03 00:31:10.177435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.177401 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:31:10.288033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.288001 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be96e00-c9a0-4c5b-9fa7-34af12668de0-kube-api-access-zvd85\" (UniqueName: \"kubernetes.io/projected/5be96e00-c9a0-4c5b-9fa7-34af12668de0-kube-api-access-zvd85\") pod \"5be96e00-c9a0-4c5b-9fa7-34af12668de0\" (UID: \"5be96e00-c9a0-4c5b-9fa7-34af12668de0\") " Nov 03 00:31:10.288205 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.288170 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be96e00-c9a0-4c5b-9fa7-34af12668de0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be96e00-c9a0-4c5b-9fa7-34af12668de0-plugins-config-volume\") pod \"5be96e00-c9a0-4c5b-9fa7-34af12668de0\" (UID: \"5be96e00-c9a0-4c5b-9fa7-34af12668de0\") " Nov 03 00:31:10.288581 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.288550 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be96e00-c9a0-4c5b-9fa7-34af12668de0-plugins-config-volume" pod "5be96e00-c9a0-4c5b-9fa7-34af12668de0" (UID: "5be96e00-c9a0-4c5b-9fa7-34af12668de0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:10.288706 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.288669 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be96e00-c9a0-4c5b-9fa7-34af12668de0-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:10.292396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.292367 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be96e00-c9a0-4c5b-9fa7-34af12668de0-kube-api-access-zvd85" pod "5be96e00-c9a0-4c5b-9fa7-34af12668de0" (UID: "5be96e00-c9a0-4c5b-9fa7-34af12668de0"). InnerVolumeSpecName "kube-api-access-zvd85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:10.390502 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.390466 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvd85\" (UniqueName: \"kubernetes.io/projected/5be96e00-c9a0-4c5b-9fa7-34af12668de0-kube-api-access-zvd85\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:10.409291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.409255 4304 generic.go:356] "Generic (PLEG): container finished" podID="5be96e00-c9a0-4c5b-9fa7-34af12668de0" containerID="cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd" exitCode=0 Nov 03 00:31:10.409483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.409300 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" event={"ID":"5be96e00-c9a0-4c5b-9fa7-34af12668de0","Type":"ContainerDied","Data":"cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd"} Nov 03 00:31:10.409483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.409318 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" Nov 03 00:31:10.409483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.409351 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz" event={"ID":"5be96e00-c9a0-4c5b-9fa7-34af12668de0","Type":"ContainerDied","Data":"bd24e86711740f5f68c2dd0764886860789b7fb61ad7474ed6d6ca5034476581"} Nov 03 00:31:10.409483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.409380 4304 scope.go:122] "RemoveContainer" containerID="cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd" Nov 03 00:31:10.420084 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.420063 4304 scope.go:122] "RemoveContainer" containerID="cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd" Nov 03 00:31:10.420359 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:10.420334 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd\": container with ID starting with cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd not found: ID does not exist" containerID="cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd" Nov 03 00:31:10.420450 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.420371 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd"} err="failed to get container status \"cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd\": rpc error: code = NotFound desc = could not find container \"cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd\": container with ID starting with cf109910b7325d5b1b143c7de60dfe89ef634510f373517c2dd86f22761058dd not found: ID does not exist" Nov 03 00:31:10.573679 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.573649 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz"] Nov 03 00:31:10.579736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.579710 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-569f76f8f7-n7tlz"] Nov 03 00:31:10.830185 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:10.830113 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be96e00-c9a0-4c5b-9fa7-34af12668de0" path="/var/lib/kubelet/pods/5be96e00-c9a0-4c5b-9fa7-34af12668de0/volumes" Nov 03 00:31:12.304404 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:12.304376 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6"] Nov 03 00:31:12.304854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:12.304580 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerName="istio-proxy" containerID="cri-o://e6043b07c2515d1fded97c9db7363494888b2757d471065bf6867e234276bed4" gracePeriod=30 Nov 03 00:31:13.601026 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:13.600994 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7"] Nov 03 00:31:13.601477 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:13.601212 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerName="istio-proxy" containerID="cri-o://257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f" gracePeriod=30 Nov 03 00:31:16.100558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.100525 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z"] Nov 03 00:31:16.101111 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101096 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerName="cleanup" Nov 03 00:31:16.101154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101112 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerName="cleanup" Nov 03 00:31:16.101154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101123 4304 cpu_manager.go:414] "containerMap: removing container" podUID="5be96e00-c9a0-4c5b-9fa7-34af12668de0" containerName="payload-processing" Nov 03 00:31:16.101154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101128 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be96e00-c9a0-4c5b-9fa7-34af12668de0" containerName="payload-processing" Nov 03 00:31:16.101154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101134 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerName="cleanup" Nov 03 00:31:16.101154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101140 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerName="cleanup" Nov 03 00:31:16.101154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101155 4304 cpu_manager.go:414] "containerMap: removing container" podUID="5b3e2a93-5326-4d2d-bfc7-77f87727a863" containerName="maas-api" Nov 03 00:31:16.101348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101159 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3e2a93-5326-4d2d-bfc7-77f87727a863" containerName="maas-api" Nov 03 00:31:16.101348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101179 4304 cpu_manager.go:414] "containerMap: removing container" podUID="3a79e336-f4c4-4d28-8023-be246046f3dc" containerName="istio-proxy" Nov 03 00:31:16.101348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101183 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a79e336-f4c4-4d28-8023-be246046f3dc" containerName="istio-proxy" Nov 03 00:31:16.101348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101262 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3e2a93-5326-4d2d-bfc7-77f87727a863" containerName="maas-api" Nov 03 00:31:16.101348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101268 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7ef0dc-c93e-4986-aaa2-4f70e7add6a1" containerName="istio-proxy" Nov 03 00:31:16.101348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101276 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a79e336-f4c4-4d28-8023-be246046f3dc" containerName="istio-proxy" Nov 03 00:31:16.101348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101282 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerName="cleanup" Nov 03 00:31:16.101348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101290 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be96e00-c9a0-4c5b-9fa7-34af12668de0" containerName="payload-processing" Nov 03 00:31:16.101348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.101295 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerName="cleanup" Nov 03 00:31:16.105962 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.105944 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.171666 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.171635 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.171863 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.171818 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-metrics-certs\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.171956 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.171935 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74fps\" (UniqueName: \"kubernetes.io/projected/d257de99-abfb-45f9-b00f-552faad901e7-kube-api-access-74fps\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.172069 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.172042 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d257de99-abfb-45f9-b00f-552faad901e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.207142 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.207119 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bb5643-serving-cert\"" Nov 03 00:31:16.210858 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.210838 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z"] Nov 03 00:31:16.273404 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.273366 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.273577 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.273459 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-metrics-certs\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.273577 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.273507 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74fps\" (UniqueName: \"kubernetes.io/projected/d257de99-abfb-45f9-b00f-552faad901e7-kube-api-access-74fps\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.273577 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:16.273525 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-serving-cert: secret "maas-api-e2e-ipp-a-bb5643-serving-cert" not found Nov 03 00:31:16.273577 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.273541 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d257de99-abfb-45f9-b00f-552faad901e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.273820 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:16.273584 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-maas-api-tls podName:d257de99-abfb-45f9-b00f-552faad901e7 nodeName:}" failed. No retries permitted until 2026-11-03 00:31:16.773569558 +0000 UTC m=+1560.057127240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-maas-api-tls") pod "maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" (UID: "d257de99-abfb-45f9-b00f-552faad901e7") : secret "maas-api-e2e-ipp-a-bb5643-serving-cert" not found Nov 03 00:31:16.274163 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.274146 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d257de99-abfb-45f9-b00f-552faad901e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.276177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.276150 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-metrics-certs\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.330150 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.330120 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74fps\" (UniqueName: \"kubernetes.io/projected/d257de99-abfb-45f9-b00f-552faad901e7-kube-api-access-74fps\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.780135 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.780108 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:16.782628 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:16.782601 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:17.018492 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.018462 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:17.399935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.399907 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z"] Nov 03 00:31:17.400707 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:31:17.400682 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd257de99_abfb_45f9_b00f_552faad901e7.slice/crio-950c31bad21c030efcc1a71e1b07707c7cc8acf19db3af1d2d444cca55ee013e WatchSource:0}: Error finding container 950c31bad21c030efcc1a71e1b07707c7cc8acf19db3af1d2d444cca55ee013e: Status 404 returned error can't find the container with id 950c31bad21c030efcc1a71e1b07707c7cc8acf19db3af1d2d444cca55ee013e Nov 03 00:31:17.409366 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.409348 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:31:17.463196 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.463049 4304 generic.go:356] "Generic (PLEG): container finished" podID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerID="e6043b07c2515d1fded97c9db7363494888b2757d471065bf6867e234276bed4" exitCode=0 Nov 03 00:31:17.463508 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.463444 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" event={"ID":"cb2a3f42-c317-4d84-9941-c12df6a7c752","Type":"ContainerDied","Data":"e6043b07c2515d1fded97c9db7363494888b2757d471065bf6867e234276bed4"} Nov 03 00:31:17.465045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.465011 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" event={"ID":"d257de99-abfb-45f9-b00f-552faad901e7","Type":"ContainerStarted","Data":"950c31bad21c030efcc1a71e1b07707c7cc8acf19db3af1d2d444cca55ee013e"} Nov 03 00:31:17.586720 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.586695 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:31:17.693014 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.692978 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-token\" (UniqueName: \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-token\") pod \"cb2a3f42-c317-4d84-9941-c12df6a7c752\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " Nov 03 00:31:17.693224 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693040 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-credential-socket\") pod \"cb2a3f42-c317-4d84-9941-c12df6a7c752\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " Nov 03 00:31:17.693224 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693088 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-podinfo\") pod \"cb2a3f42-c317-4d84-9941-c12df6a7c752\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " Nov 03 00:31:17.693224 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693134 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-data\") pod \"cb2a3f42-c317-4d84-9941-c12df6a7c752\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " Nov 03 00:31:17.693224 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693211 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-socket\") pod \"cb2a3f42-c317-4d84-9941-c12df6a7c752\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " Nov 03 00:31:17.693444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693260 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-certs\") pod \"cb2a3f42-c317-4d84-9941-c12df6a7c752\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " Nov 03 00:31:17.693444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693321 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-kube-api-access-hvzhc\" (UniqueName: \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-kube-api-access-hvzhc\") pod \"cb2a3f42-c317-4d84-9941-c12df6a7c752\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " Nov 03 00:31:17.693444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693372 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-credential-socket" pod "cb2a3f42-c317-4d84-9941-c12df6a7c752" (UID: "cb2a3f42-c317-4d84-9941-c12df6a7c752"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.693444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693383 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb2a3f42-c317-4d84-9941-c12df6a7c752-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb2a3f42-c317-4d84-9941-c12df6a7c752-istiod-ca-cert\") pod \"cb2a3f42-c317-4d84-9941-c12df6a7c752\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " Nov 03 00:31:17.693641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693481 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-data" pod "cb2a3f42-c317-4d84-9941-c12df6a7c752" (UID: "cb2a3f42-c317-4d84-9941-c12df6a7c752"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.693641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693502 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-envoy\") pod \"cb2a3f42-c317-4d84-9941-c12df6a7c752\" (UID: \"cb2a3f42-c317-4d84-9941-c12df6a7c752\") " Nov 03 00:31:17.693641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693502 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-socket" pod "cb2a3f42-c317-4d84-9941-c12df6a7c752" (UID: "cb2a3f42-c317-4d84-9941-c12df6a7c752"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.693641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693591 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-certs" pod "cb2a3f42-c317-4d84-9941-c12df6a7c752" (UID: "cb2a3f42-c317-4d84-9941-c12df6a7c752"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.694050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.693668 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2a3f42-c317-4d84-9941-c12df6a7c752-istiod-ca-cert" pod "cb2a3f42-c317-4d84-9941-c12df6a7c752" (UID: "cb2a3f42-c317-4d84-9941-c12df6a7c752"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:17.694119 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.694099 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.694265 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.694195 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.694327 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.694298 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb2a3f42-c317-4d84-9941-c12df6a7c752-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.694327 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.694313 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.694441 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.694326 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.695539 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.695509 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-podinfo" pod "cb2a3f42-c317-4d84-9941-c12df6a7c752" (UID: "cb2a3f42-c317-4d84-9941-c12df6a7c752"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:17.695655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.695621 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-kube-api-access-hvzhc" pod "cb2a3f42-c317-4d84-9941-c12df6a7c752" (UID: "cb2a3f42-c317-4d84-9941-c12df6a7c752"). InnerVolumeSpecName "kube-api-access-hvzhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:17.695711 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.695654 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-token" pod "cb2a3f42-c317-4d84-9941-c12df6a7c752" (UID: "cb2a3f42-c317-4d84-9941-c12df6a7c752"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:17.695711 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.695693 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-envoy" pod "cb2a3f42-c317-4d84-9941-c12df6a7c752" (UID: "cb2a3f42-c317-4d84-9941-c12df6a7c752"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.796178 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.796142 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.796178 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.796174 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvzhc\" (UniqueName: \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-kube-api-access-hvzhc\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.796178 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.796185 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.796479 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:17.796193 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb2a3f42-c317-4d84-9941-c12df6a7c752-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:18.483965 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:18.482569 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" event={"ID":"d257de99-abfb-45f9-b00f-552faad901e7","Type":"ContainerStarted","Data":"aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9"} Nov 03 00:31:18.483965 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:18.483859 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:18.489042 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:18.486217 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" event={"ID":"cb2a3f42-c317-4d84-9941-c12df6a7c752","Type":"ContainerDied","Data":"69db29582465ad5eb1d92b6cb64d3b3613549fe97b6e41bc665e36aa5cedbe0d"} Nov 03 00:31:18.489042 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:18.486273 4304 scope.go:122] "RemoveContainer" containerID="e6043b07c2515d1fded97c9db7363494888b2757d471065bf6867e234276bed4" Nov 03 00:31:18.489042 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:18.486422 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6" Nov 03 00:31:18.600014 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:18.599916 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" podStartSLOduration=1.744465313 podStartE2EDuration="2.599901496s" totalImagesPullingTime="855.436183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:18.598254565 +0000 UTC m=+1561.881812238" watchObservedRunningTime="2026-11-03 00:31:18.599901496 +0000 UTC m=+1561.883459169" Nov 03 00:31:18.724598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:18.722711 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6"] Nov 03 00:31:18.728195 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:18.728164 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-vs8x6"] Nov 03 00:31:18.833100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:18.833067 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" path="/var/lib/kubelet/pods/cb2a3f42-c317-4d84-9941-c12df6a7c752/volumes" Nov 03 00:31:18.901023 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:18.901001 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:31:19.040340 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.040305 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-podinfo\") pod \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " Nov 03 00:31:19.040525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.040461 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-kube-api-access-vvjf8\" (UniqueName: \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-kube-api-access-vvjf8\") pod \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " Nov 03 00:31:19.040525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.040494 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-credential-socket\") pod \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " Nov 03 00:31:19.040729 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.040699 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-certs\") pod \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " Nov 03 00:31:19.041216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.040819 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-data\") pod \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " Nov 03 00:31:19.041216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.040855 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-credential-socket" pod "31b6c870-bf3c-4bd1-8ffb-acf522c710d9" (UID: "31b6c870-bf3c-4bd1-8ffb-acf522c710d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:19.041216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.040906 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-socket\") pod \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " Nov 03 00:31:19.041216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.041028 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-certs" pod "31b6c870-bf3c-4bd1-8ffb-acf522c710d9" (UID: "31b6c870-bf3c-4bd1-8ffb-acf522c710d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:19.041216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.041198 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-socket" pod "31b6c870-bf3c-4bd1-8ffb-acf522c710d9" (UID: "31b6c870-bf3c-4bd1-8ffb-acf522c710d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:19.041898 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.041282 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-data" pod "31b6c870-bf3c-4bd1-8ffb-acf522c710d9" (UID: "31b6c870-bf3c-4bd1-8ffb-acf522c710d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:19.041898 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.041336 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istiod-ca-cert\") pod \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " Nov 03 00:31:19.041898 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.041420 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-envoy\") pod \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " Nov 03 00:31:19.041898 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.041459 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-token\" (UniqueName: \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-token\") pod \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\" (UID: \"31b6c870-bf3c-4bd1-8ffb-acf522c710d9\") " Nov 03 00:31:19.042239 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.042035 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:19.042239 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.042055 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:19.042239 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.042069 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:19.042239 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.042084 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:19.043137 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.042605 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istiod-ca-cert" pod "31b6c870-bf3c-4bd1-8ffb-acf522c710d9" (UID: "31b6c870-bf3c-4bd1-8ffb-acf522c710d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:19.044331 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.044307 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-kube-api-access-vvjf8" pod "31b6c870-bf3c-4bd1-8ffb-acf522c710d9" (UID: "31b6c870-bf3c-4bd1-8ffb-acf522c710d9"). InnerVolumeSpecName "kube-api-access-vvjf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:19.044679 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.044652 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-podinfo" pod "31b6c870-bf3c-4bd1-8ffb-acf522c710d9" (UID: "31b6c870-bf3c-4bd1-8ffb-acf522c710d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:19.044984 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.044953 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-token" pod "31b6c870-bf3c-4bd1-8ffb-acf522c710d9" (UID: "31b6c870-bf3c-4bd1-8ffb-acf522c710d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:19.045095 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.045026 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-envoy" pod "31b6c870-bf3c-4bd1-8ffb-acf522c710d9" (UID: "31b6c870-bf3c-4bd1-8ffb-acf522c710d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:19.143185 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.143160 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:19.143328 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.143189 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:19.143328 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.143199 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:19.143328 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.143208 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvjf8\" (UniqueName: \"kubernetes.io/projected/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-kube-api-access-vvjf8\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:19.143328 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.143220 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b6c870-bf3c-4bd1-8ffb-acf522c710d9-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:19.495715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.495611 4304 generic.go:356] "Generic (PLEG): container finished" podID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerID="257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f" exitCode=0 Nov 03 00:31:19.495715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.495664 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" Nov 03 00:31:19.495715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.495697 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" event={"ID":"31b6c870-bf3c-4bd1-8ffb-acf522c710d9","Type":"ContainerDied","Data":"257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f"} Nov 03 00:31:19.496345 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.495758 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7" event={"ID":"31b6c870-bf3c-4bd1-8ffb-acf522c710d9","Type":"ContainerDied","Data":"0b289fd2eba0b8337abdee78185c811ad3467d7d02516c051c1fccf0a98cd9de"} Nov 03 00:31:19.496345 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.495819 4304 scope.go:122] "RemoveContainer" containerID="257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f" Nov 03 00:31:19.507060 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.507040 4304 scope.go:122] "RemoveContainer" containerID="257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f" Nov 03 00:31:19.507339 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:19.507323 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f\": container with ID starting with 257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f not found: ID does not exist" containerID="257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f" Nov 03 00:31:19.507385 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.507351 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f"} err="failed to get container status \"257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f\": rpc error: code = NotFound desc = could not find container \"257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f\": container with ID starting with 257bad3017b3121960c37fa843b9075be179437f4d30be8bad34077515b95c3f not found: ID does not exist" Nov 03 00:31:19.662179 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.662149 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7"] Nov 03 00:31:19.712189 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.712160 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-dljv7"] Nov 03 00:31:19.715101 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:19.715077 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6"] Nov 03 00:31:20.830388 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:20.830355 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" path="/var/lib/kubelet/pods/31b6c870-bf3c-4bd1-8ffb-acf522c710d9/volumes" Nov 03 00:31:24.179287 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.179241 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w"] Nov 03 00:31:24.180853 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.180824 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerName="cleanup" Nov 03 00:31:24.181032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.181016 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerName="cleanup" Nov 03 00:31:24.181135 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.181123 4304 cpu_manager.go:414] "containerMap: removing container" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerName="istio-proxy" Nov 03 00:31:24.181214 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.181204 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerName="istio-proxy" Nov 03 00:31:24.181336 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.181325 4304 cpu_manager.go:414] "containerMap: removing container" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerName="istio-proxy" Nov 03 00:31:24.181411 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.181401 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerName="istio-proxy" Nov 03 00:31:24.181647 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.181631 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b6c870-bf3c-4bd1-8ffb-acf522c710d9" containerName="istio-proxy" Nov 03 00:31:24.181768 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.181756 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6326520-8f7d-425a-86e4-ba63ece9e79a" containerName="cleanup" Nov 03 00:31:24.181883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.181870 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2a3f42-c317-4d84-9941-c12df6a7c752" containerName="istio-proxy" Nov 03 00:31:24.197827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.197801 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:31:24.203296 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.203265 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w"] Nov 03 00:31:24.299329 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.299297 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3df441\"" Nov 03 00:31:24.299329 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.299311 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3df441-dockercfg-4q7mm\"" Nov 03 00:31:24.343899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.343867 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0622afbb-ae90-40fe-b918-910e6a035f85-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3df441-774876f949-94z5w\" (UID: \"0622afbb-ae90-40fe-b918-910e6a035f85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:31:24.343899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.343898 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjmc\" (UniqueName: \"kubernetes.io/projected/0622afbb-ae90-40fe-b918-910e6a035f85-kube-api-access-9zjmc\") pod \"payload-processing-e2e-ipp-a-3df441-774876f949-94z5w\" (UID: \"0622afbb-ae90-40fe-b918-910e6a035f85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:31:24.445124 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.445046 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0622afbb-ae90-40fe-b918-910e6a035f85-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3df441-774876f949-94z5w\" (UID: \"0622afbb-ae90-40fe-b918-910e6a035f85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:31:24.445124 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.445082 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjmc\" (UniqueName: \"kubernetes.io/projected/0622afbb-ae90-40fe-b918-910e6a035f85-kube-api-access-9zjmc\") pod \"payload-processing-e2e-ipp-a-3df441-774876f949-94z5w\" (UID: \"0622afbb-ae90-40fe-b918-910e6a035f85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:31:24.445694 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.445676 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0622afbb-ae90-40fe-b918-910e6a035f85-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3df441-774876f949-94z5w\" (UID: \"0622afbb-ae90-40fe-b918-910e6a035f85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:31:24.502899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.502868 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjmc\" (UniqueName: \"kubernetes.io/projected/0622afbb-ae90-40fe-b918-910e6a035f85-kube-api-access-9zjmc\") pod \"payload-processing-e2e-ipp-a-3df441-774876f949-94z5w\" (UID: \"0622afbb-ae90-40fe-b918-910e6a035f85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:31:24.524668 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.524643 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:31:24.715553 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.715444 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" podUID="b9315860-a8bf-4d57-b752-2170be50dd40" containerName="payload-processing" containerID="cri-o://c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e" gracePeriod=25 Nov 03 00:31:24.901631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.901577 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w"] Nov 03 00:31:24.976222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:24.976190 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:31:25.052228 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.052195 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9315860-a8bf-4d57-b752-2170be50dd40-kube-api-access-vjkjp\" (UniqueName: \"kubernetes.io/projected/b9315860-a8bf-4d57-b752-2170be50dd40-kube-api-access-vjkjp\") pod \"b9315860-a8bf-4d57-b752-2170be50dd40\" (UID: \"b9315860-a8bf-4d57-b752-2170be50dd40\") " Nov 03 00:31:25.052388 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.052284 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9315860-a8bf-4d57-b752-2170be50dd40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9315860-a8bf-4d57-b752-2170be50dd40-plugins-config-volume\") pod \"b9315860-a8bf-4d57-b752-2170be50dd40\" (UID: \"b9315860-a8bf-4d57-b752-2170be50dd40\") " Nov 03 00:31:25.052624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.052601 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9315860-a8bf-4d57-b752-2170be50dd40-plugins-config-volume" pod "b9315860-a8bf-4d57-b752-2170be50dd40" (UID: "b9315860-a8bf-4d57-b752-2170be50dd40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:25.054385 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.054366 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9315860-a8bf-4d57-b752-2170be50dd40-kube-api-access-vjkjp" pod "b9315860-a8bf-4d57-b752-2170be50dd40" (UID: "b9315860-a8bf-4d57-b752-2170be50dd40"). InnerVolumeSpecName "kube-api-access-vjkjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:25.153758 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.153725 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjkjp\" (UniqueName: \"kubernetes.io/projected/b9315860-a8bf-4d57-b752-2170be50dd40-kube-api-access-vjkjp\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:25.153758 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.153755 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9315860-a8bf-4d57-b752-2170be50dd40-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:25.509768 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.509739 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:31:25.543059 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.543022 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" event={"ID":"0622afbb-ae90-40fe-b918-910e6a035f85","Type":"ContainerStarted","Data":"92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4"} Nov 03 00:31:25.543227 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.543081 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" event={"ID":"0622afbb-ae90-40fe-b918-910e6a035f85","Type":"ContainerStarted","Data":"c37431a17b89ba52cc684881eae09b04d3a69e7f0ad6fd72fda0a74035cbdb6c"} Nov 03 00:31:25.543227 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.543104 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:31:25.544337 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.544315 4304 generic.go:356] "Generic (PLEG): container finished" podID="b9315860-a8bf-4d57-b752-2170be50dd40" containerID="c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e" exitCode=0 Nov 03 00:31:25.544424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.544353 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" Nov 03 00:31:25.544459 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.544352 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" event={"ID":"b9315860-a8bf-4d57-b752-2170be50dd40","Type":"ContainerDied","Data":"c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e"} Nov 03 00:31:25.544490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.544468 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6" event={"ID":"b9315860-a8bf-4d57-b752-2170be50dd40","Type":"ContainerDied","Data":"c507d011f435c46297788e085cf803138834a89ab302134d3c5c4b2b84cc6444"} Nov 03 00:31:25.544532 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.544497 4304 scope.go:122] "RemoveContainer" containerID="c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e" Nov 03 00:31:25.553992 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.553975 4304 scope.go:122] "RemoveContainer" containerID="c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e" Nov 03 00:31:25.554286 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:25.554256 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e\": container with ID starting with c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e not found: ID does not exist" containerID="c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e" Nov 03 00:31:25.554372 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.554285 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e"} err="failed to get container status \"c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e\": rpc error: code = NotFound desc = could not find container \"c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e\": container with ID starting with c8b58f9a834c864a38f7c55b13b05814e07d60fc5d647b24398f13e4a43c5c1e not found: ID does not exist" Nov 03 00:31:25.736279 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.736223 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" podStartSLOduration=1.736204115 podStartE2EDuration="1.736204115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:25.735487527 +0000 UTC m=+1569.019045202" watchObservedRunningTime="2026-11-03 00:31:25.736204115 +0000 UTC m=+1569.019761790" Nov 03 00:31:25.863417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.861916 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6"] Nov 03 00:31:25.878318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:25.878289 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-be578d-6ffbf5cb57-lfcw6"] Nov 03 00:31:26.829831 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:26.829796 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9315860-a8bf-4d57-b752-2170be50dd40" path="/var/lib/kubelet/pods/b9315860-a8bf-4d57-b752-2170be50dd40/volumes" Nov 03 00:31:28.655543 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:28.655507 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:31:28.684256 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:28.684222 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58"] Nov 03 00:31:28.684564 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:28.684535 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" podUID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" containerName="istio-proxy" containerID="cri-o://10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824" gracePeriod=30 Nov 03 00:31:33.952170 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:33.952113 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:31:34.066872 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.066845 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-token\" (UniqueName: \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-token\") pod \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " Nov 03 00:31:34.067040 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.066917 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-data\") pod \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " Nov 03 00:31:34.067083 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067036 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-kube-api-access-phq4t\" (UniqueName: \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-kube-api-access-phq4t\") pod \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " Nov 03 00:31:34.067083 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067064 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istiod-ca-cert\") pod \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " Nov 03 00:31:34.067169 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067148 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-envoy\") pod \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " Nov 03 00:31:34.067230 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067179 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-certs\") pod \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " Nov 03 00:31:34.067230 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067214 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-data" pod "9b2cf84f-1a22-49da-80a3-56ce09040f1c" (UID: "9b2cf84f-1a22-49da-80a3-56ce09040f1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.067336 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067257 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-podinfo\") pod \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " Nov 03 00:31:34.067336 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067298 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-credential-socket\") pod \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " Nov 03 00:31:34.067446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067370 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-socket\") pod \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\" (UID: \"9b2cf84f-1a22-49da-80a3-56ce09040f1c\") " Nov 03 00:31:34.067446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067409 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-certs" pod "9b2cf84f-1a22-49da-80a3-56ce09040f1c" (UID: "9b2cf84f-1a22-49da-80a3-56ce09040f1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.067550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067499 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istiod-ca-cert" pod "9b2cf84f-1a22-49da-80a3-56ce09040f1c" (UID: "9b2cf84f-1a22-49da-80a3-56ce09040f1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:34.067742 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067714 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-credential-socket" pod "9b2cf84f-1a22-49da-80a3-56ce09040f1c" (UID: "9b2cf84f-1a22-49da-80a3-56ce09040f1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.067918 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.067888 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-socket" pod "9b2cf84f-1a22-49da-80a3-56ce09040f1c" (UID: "9b2cf84f-1a22-49da-80a3-56ce09040f1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.068091 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.068072 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.068174 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.068096 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.068174 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.068110 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.068174 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.068124 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.068174 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.068137 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.069316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.069295 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-token" pod "9b2cf84f-1a22-49da-80a3-56ce09040f1c" (UID: "9b2cf84f-1a22-49da-80a3-56ce09040f1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:34.069529 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.069503 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-podinfo" pod "9b2cf84f-1a22-49da-80a3-56ce09040f1c" (UID: "9b2cf84f-1a22-49da-80a3-56ce09040f1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:34.069719 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.069701 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-kube-api-access-phq4t" pod "9b2cf84f-1a22-49da-80a3-56ce09040f1c" (UID: "9b2cf84f-1a22-49da-80a3-56ce09040f1c"). InnerVolumeSpecName "kube-api-access-phq4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:34.069763 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.069700 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-envoy" pod "9b2cf84f-1a22-49da-80a3-56ce09040f1c" (UID: "9b2cf84f-1a22-49da-80a3-56ce09040f1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.169758 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.169719 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.169758 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.169758 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phq4t\" (UniqueName: \"kubernetes.io/projected/9b2cf84f-1a22-49da-80a3-56ce09040f1c-kube-api-access-phq4t\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.170011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.169806 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.170011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.169821 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2cf84f-1a22-49da-80a3-56ce09040f1c-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.603553 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.603520 4304 generic.go:356] "Generic (PLEG): container finished" podID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" containerID="10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824" exitCode=0 Nov 03 00:31:34.603717 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.603583 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" Nov 03 00:31:34.603717 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.603594 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" event={"ID":"9b2cf84f-1a22-49da-80a3-56ce09040f1c","Type":"ContainerDied","Data":"10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824"} Nov 03 00:31:34.603717 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.603668 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58" event={"ID":"9b2cf84f-1a22-49da-80a3-56ce09040f1c","Type":"ContainerDied","Data":"7a4d00ebfc30f01f2a57fa0c5998a212e86a1c9cf0cbcfe7370b894a322a1573"} Nov 03 00:31:34.603717 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.603689 4304 scope.go:122] "RemoveContainer" containerID="10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824" Nov 03 00:31:34.614597 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.614577 4304 scope.go:122] "RemoveContainer" containerID="10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824" Nov 03 00:31:34.614876 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:34.614857 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824\": container with ID starting with 10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824 not found: ID does not exist" containerID="10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824" Nov 03 00:31:34.614944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.614885 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824"} err="failed to get container status \"10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824\": rpc error: code = NotFound desc = could not find container \"10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824\": container with ID starting with 10eb4e1cf61b535dacf5d8b471e634eb4406d9a853175762a679c57e6ea42824 not found: ID does not exist" Nov 03 00:31:34.771343 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.771306 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58"] Nov 03 00:31:34.820523 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.820491 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-fvk58"] Nov 03 00:31:34.831047 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:34.831019 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" path="/var/lib/kubelet/pods/9b2cf84f-1a22-49da-80a3-56ce09040f1c/volumes" Nov 03 00:31:36.555130 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:36.555102 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:31:42.647608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:42.647565 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76"] Nov 03 00:31:42.648060 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:42.647858 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" containerID="cri-o://df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b" gracePeriod=30 Nov 03 00:31:42.688910 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:42.688877 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5"] Nov 03 00:31:42.689113 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:42.689093 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" containerID="cri-o://b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff" gracePeriod=30 Nov 03 00:31:42.691633 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:42.691609 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94"] Nov 03 00:31:42.691844 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:42.691816 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" containerID="cri-o://73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb" gracePeriod=30 Nov 03 00:31:42.748123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:42.748093 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf"] Nov 03 00:31:42.748323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:42.748300 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerName="istio-proxy" containerID="cri-o://51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84" gracePeriod=30 Nov 03 00:31:46.570339 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:46.570302 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:31:46.570765 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:46.570358 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:31:46.714405 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:46.714327 4304 generic.go:356] "Generic (PLEG): container finished" podID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerID="05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759" exitCode=0 Nov 03 00:31:46.714549 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:46.714401 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" event={"ID":"c2d8924d-ef4a-4891-9492-b257cbdc7647","Type":"ContainerDied","Data":"05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759"} Nov 03 00:31:46.714856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:46.714839 4304 scope.go:122] "RemoveContainer" containerID="05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759" Nov 03 00:31:46.716318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:46.716286 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:31:46.716410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:46.716363 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:31:46.716410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:46.716397 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:31:47.747011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:47.740330 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" event={"ID":"c2d8924d-ef4a-4891-9492-b257cbdc7647","Type":"ContainerStarted","Data":"f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678"} Nov 03 00:31:48.062067 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.061871 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:31:48.113833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.113803 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:31:48.114032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.114008 4304 patch_prober.go:29] interesting pod/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:31:48.114125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.114069 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 00:31:48.159799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.151725 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:31:48.171150 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171120 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-credential-socket\") pod \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " Nov 03 00:31:48.171321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171169 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-socket\") pod \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " Nov 03 00:31:48.171321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171217 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-podinfo\") pod \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " Nov 03 00:31:48.171321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171262 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-credential-socket\") pod \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " Nov 03 00:31:48.171321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171303 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-kube-api-access-fvkw6\" (UniqueName: \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-kube-api-access-fvkw6\") pod \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " Nov 03 00:31:48.171528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171337 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istiod-ca-cert\") pod \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " Nov 03 00:31:48.171528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171403 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-certs\") pod \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " Nov 03 00:31:48.171528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171417 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-credential-socket" pod "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" (UID: "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.171528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171439 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-certs\") pod \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " Nov 03 00:31:48.171528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171466 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-podinfo\") pod \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " Nov 03 00:31:48.171528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171496 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-kube-api-access-gtgnc\" (UniqueName: \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-kube-api-access-gtgnc\") pod \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " Nov 03 00:31:48.171528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171525 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-data\") pod \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " Nov 03 00:31:48.171886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171541 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-token\" (UniqueName: \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-token\") pod \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " Nov 03 00:31:48.171886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171597 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-data\") pod \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " Nov 03 00:31:48.171886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171674 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-envoy\") pod \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\" (UID: \"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c\") " Nov 03 00:31:48.171886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171724 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-token\" (UniqueName: \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-token\") pod \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " Nov 03 00:31:48.171886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171794 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-socket\") pod \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " Nov 03 00:31:48.171886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171825 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-envoy\") pod \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " Nov 03 00:31:48.171886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171839 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istiod-ca-cert" pod "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" (UID: "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:48.172554 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171892 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istiod-ca-cert\") pod \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\" (UID: \"f456d8be-bb77-48f5-83fa-f684bcbc6cb9\") " Nov 03 00:31:48.172554 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.171952 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-certs" pod "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" (UID: "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.172554 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.172187 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-socket" pod "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" (UID: "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.172554 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.172443 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-certs" pod "f456d8be-bb77-48f5-83fa-f684bcbc6cb9" (UID: "f456d8be-bb77-48f5-83fa-f684bcbc6cb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.172554 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.172523 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istiod-ca-cert" pod "f456d8be-bb77-48f5-83fa-f684bcbc6cb9" (UID: "f456d8be-bb77-48f5-83fa-f684bcbc6cb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:48.172844 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.172819 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-socket" pod "f456d8be-bb77-48f5-83fa-f684bcbc6cb9" (UID: "f456d8be-bb77-48f5-83fa-f684bcbc6cb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.172903 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.172875 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.172903 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.172898 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.173002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.172912 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.173002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.172927 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.173002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.172941 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.173002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.172956 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.173191 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.173065 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-credential-socket" pod "f456d8be-bb77-48f5-83fa-f684bcbc6cb9" (UID: "f456d8be-bb77-48f5-83fa-f684bcbc6cb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.173476 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.173443 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-data" pod "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" (UID: "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.173615 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.173577 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-data" pod "f456d8be-bb77-48f5-83fa-f684bcbc6cb9" (UID: "f456d8be-bb77-48f5-83fa-f684bcbc6cb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.175660 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.175601 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-podinfo" pod "f456d8be-bb77-48f5-83fa-f684bcbc6cb9" (UID: "f456d8be-bb77-48f5-83fa-f684bcbc6cb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:48.177154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.177127 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-podinfo" pod "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" (UID: "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:48.178401 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.178362 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-kube-api-access-gtgnc" pod "f456d8be-bb77-48f5-83fa-f684bcbc6cb9" (UID: "f456d8be-bb77-48f5-83fa-f684bcbc6cb9"). InnerVolumeSpecName "kube-api-access-gtgnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:48.178401 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.178380 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-token" pod "f456d8be-bb77-48f5-83fa-f684bcbc6cb9" (UID: "f456d8be-bb77-48f5-83fa-f684bcbc6cb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:48.178550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.178412 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-token" pod "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" (UID: "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:48.178966 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.178934 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-envoy" pod "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" (UID: "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.179048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.179028 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-kube-api-access-fvkw6" pod "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" (UID: "cb8636cf-f4f7-440d-86a4-fb4dc5aa771c"). InnerVolumeSpecName "kube-api-access-fvkw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:48.181187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.181166 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-envoy" pod "f456d8be-bb77-48f5-83fa-f684bcbc6cb9" (UID: "f456d8be-bb77-48f5-83fa-f684bcbc6cb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.217754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.217725 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:31:48.219894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.219820 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:31:48.273550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273457 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-envoy\") pod \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " Nov 03 00:31:48.273550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273546 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-kube-api-access-c8gz2\" (UniqueName: \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-kube-api-access-c8gz2\") pod \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " Nov 03 00:31:48.273766 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273574 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-credential-socket\") pod \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " Nov 03 00:31:48.273766 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273598 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-data\") pod \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " Nov 03 00:31:48.273766 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273619 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-data\") pod \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " Nov 03 00:31:48.273766 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273639 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istiod-ca-cert\") pod \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " Nov 03 00:31:48.273766 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273666 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-podinfo\") pod \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " Nov 03 00:31:48.274057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273871 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-certs\") pod \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " Nov 03 00:31:48.274057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273937 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-credential-socket\") pod \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " Nov 03 00:31:48.274057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273964 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-podinfo\") pod \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " Nov 03 00:31:48.274057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273927 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-data" pod "054e4951-35e0-44e9-a75d-a6d4cb457b1d" (UID: "054e4951-35e0-44e9-a75d-a6d4cb457b1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.274057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273943 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-data" pod "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" (UID: "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.274057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.273974 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-credential-socket" pod "054e4951-35e0-44e9-a75d-a6d4cb457b1d" (UID: "054e4951-35e0-44e9-a75d-a6d4cb457b1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.274057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274028 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-socket\") pod \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " Nov 03 00:31:48.274057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274057 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-token\" (UniqueName: \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-token\") pod \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " Nov 03 00:31:48.274467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274087 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-certs\") pod \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " Nov 03 00:31:48.274467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274108 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istiod-ca-cert\") pod \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " Nov 03 00:31:48.274467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274127 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-envoy\") pod \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " Nov 03 00:31:48.274467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274146 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-kube-api-access-4gqmz\" (UniqueName: \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-kube-api-access-4gqmz\") pod \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\" (UID: \"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae\") " Nov 03 00:31:48.274467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274145 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-credential-socket" pod "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" (UID: "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.274467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274169 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-socket\") pod \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " Nov 03 00:31:48.274467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274189 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-certs" pod "054e4951-35e0-44e9-a75d-a6d4cb457b1d" (UID: "054e4951-35e0-44e9-a75d-a6d4cb457b1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.274467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274199 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-token\" (UniqueName: \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-token\") pod \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\" (UID: \"054e4951-35e0-44e9-a75d-a6d4cb457b1d\") " Nov 03 00:31:48.274467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274354 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-certs" pod "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" (UID: "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.274944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274506 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-socket" pod "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" (UID: "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.274944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274576 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istiod-ca-cert" pod "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" (UID: "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:48.274944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274846 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.274944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274868 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.274944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274884 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtgnc\" (UniqueName: \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-kube-api-access-gtgnc\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.274944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274898 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.274944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274913 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.274944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274928 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.274944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274942 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274955 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274970 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274985 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.274999 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.275011 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.275026 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.275041 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.275053 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.275066 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.275079 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.275094 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f456d8be-bb77-48f5-83fa-f684bcbc6cb9-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.275108 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvkw6\" (UniqueName: \"kubernetes.io/projected/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c-kube-api-access-fvkw6\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.275432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.275123 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.276181 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.275804 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istiod-ca-cert" pod "054e4951-35e0-44e9-a75d-a6d4cb457b1d" (UID: "054e4951-35e0-44e9-a75d-a6d4cb457b1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:48.276960 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.276929 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-podinfo" pod "054e4951-35e0-44e9-a75d-a6d4cb457b1d" (UID: "054e4951-35e0-44e9-a75d-a6d4cb457b1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:48.277088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.276966 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-kube-api-access-c8gz2" pod "054e4951-35e0-44e9-a75d-a6d4cb457b1d" (UID: "054e4951-35e0-44e9-a75d-a6d4cb457b1d"). InnerVolumeSpecName "kube-api-access-c8gz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:48.277088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.276991 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-token" pod "054e4951-35e0-44e9-a75d-a6d4cb457b1d" (UID: "054e4951-35e0-44e9-a75d-a6d4cb457b1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:48.277198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.277063 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-envoy" pod "054e4951-35e0-44e9-a75d-a6d4cb457b1d" (UID: "054e4951-35e0-44e9-a75d-a6d4cb457b1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.277198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.277160 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-socket" pod "054e4951-35e0-44e9-a75d-a6d4cb457b1d" (UID: "054e4951-35e0-44e9-a75d-a6d4cb457b1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.277306 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.277218 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-podinfo" pod "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" (UID: "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:48.277306 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.277258 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-token" pod "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" (UID: "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:48.278507 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.278467 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-envoy" pod "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" (UID: "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:48.278954 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.278928 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-kube-api-access-4gqmz" pod "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" (UID: "4dd0d831-a1b5-4ff0-ab36-726b1c1762ae"). InnerVolumeSpecName "kube-api-access-4gqmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:48.376584 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.376551 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.376584 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.376580 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.376584 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.376590 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.376856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.376599 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gqmz\" (UniqueName: \"kubernetes.io/projected/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-kube-api-access-4gqmz\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.376856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.376610 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.376856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.376618 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.376856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.376627 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.376856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.376636 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8gz2\" (UniqueName: \"kubernetes.io/projected/054e4951-35e0-44e9-a75d-a6d4cb457b1d-kube-api-access-c8gz2\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.376856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.376643 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054e4951-35e0-44e9-a75d-a6d4cb457b1d-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.376856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.376651 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:48.751434 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.751395 4304 generic.go:356] "Generic (PLEG): container finished" podID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerID="73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb" exitCode=0 Nov 03 00:31:48.751951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.751539 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" event={"ID":"054e4951-35e0-44e9-a75d-a6d4cb457b1d","Type":"ContainerDied","Data":"73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb"} Nov 03 00:31:48.751951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.751536 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" Nov 03 00:31:48.751951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.751604 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94" event={"ID":"054e4951-35e0-44e9-a75d-a6d4cb457b1d","Type":"ContainerDied","Data":"834f56a76fe11573a7eafb7b6fe24a84a0488ba813c252fdb07b5e54056a664b"} Nov 03 00:31:48.751951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.751634 4304 scope.go:122] "RemoveContainer" containerID="73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb" Nov 03 00:31:48.753442 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.753413 4304 generic.go:356] "Generic (PLEG): container finished" podID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerID="b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff" exitCode=0 Nov 03 00:31:48.753586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.753518 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" Nov 03 00:31:48.753586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.753539 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" event={"ID":"f456d8be-bb77-48f5-83fa-f684bcbc6cb9","Type":"ContainerDied","Data":"b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff"} Nov 03 00:31:48.753696 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.753600 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5" event={"ID":"f456d8be-bb77-48f5-83fa-f684bcbc6cb9","Type":"ContainerDied","Data":"d127feefe47b3ff2e7deedb18a9ec6eee0f3c47167df7808ceb659574f5bb3c1"} Nov 03 00:31:48.755357 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.755335 4304 generic.go:356] "Generic (PLEG): container finished" podID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerID="df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b" exitCode=0 Nov 03 00:31:48.755488 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.755384 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" Nov 03 00:31:48.755488 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.755397 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" event={"ID":"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c","Type":"ContainerDied","Data":"df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b"} Nov 03 00:31:48.755488 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.755448 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76" event={"ID":"cb8636cf-f4f7-440d-86a4-fb4dc5aa771c","Type":"ContainerDied","Data":"1a6fbf0ae667ccbd345da4c42bba024bb8d055485adbbefa2156fd1c8e6da8e0"} Nov 03 00:31:48.757069 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.757046 4304 generic.go:356] "Generic (PLEG): container finished" podID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerID="51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84" exitCode=0 Nov 03 00:31:48.757328 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.757304 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" event={"ID":"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae","Type":"ContainerDied","Data":"51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84"} Nov 03 00:31:48.757408 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.757353 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" event={"ID":"4dd0d831-a1b5-4ff0-ab36-726b1c1762ae","Type":"ContainerDied","Data":"ddc8c71cb1fa6b864c16de5fcdc4d56d414aa52e11c6b3e0e786d1c9fb7c45cf"} Nov 03 00:31:48.757467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.757421 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf" Nov 03 00:31:48.766557 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.766540 4304 scope.go:122] "RemoveContainer" containerID="73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb" Nov 03 00:31:48.766886 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:48.766866 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb\": container with ID starting with 73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb not found: ID does not exist" containerID="73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb" Nov 03 00:31:48.766981 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.766891 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb"} err="failed to get container status \"73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb\": rpc error: code = NotFound desc = could not find container \"73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb\": container with ID starting with 73275ecae2738f1fdc3c36390fb95e2b47b351f5ff7fa768a0bf0ce785c4e8fb not found: ID does not exist" Nov 03 00:31:48.766981 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.766909 4304 scope.go:122] "RemoveContainer" containerID="b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff" Nov 03 00:31:48.776469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.776449 4304 scope.go:122] "RemoveContainer" containerID="b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff" Nov 03 00:31:48.776716 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:48.776700 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff\": container with ID starting with b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff not found: ID does not exist" containerID="b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff" Nov 03 00:31:48.776797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.776723 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff"} err="failed to get container status \"b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff\": rpc error: code = NotFound desc = could not find container \"b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff\": container with ID starting with b2216ef039d3135742ac90e45c8b31aec61c513650726fe075f2f34186831aff not found: ID does not exist" Nov 03 00:31:48.776797 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.776748 4304 scope.go:122] "RemoveContainer" containerID="df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b" Nov 03 00:31:48.785099 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.785084 4304 scope.go:122] "RemoveContainer" containerID="df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b" Nov 03 00:31:48.785338 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:48.785322 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b\": container with ID starting with df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b not found: ID does not exist" containerID="df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b" Nov 03 00:31:48.785384 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.785344 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b"} err="failed to get container status \"df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b\": rpc error: code = NotFound desc = could not find container \"df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b\": container with ID starting with df241a4b1a2fc26d64e3103d17086123be7f6837cea01dc942ba2155725f2e6b not found: ID does not exist" Nov 03 00:31:48.785384 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.785359 4304 scope.go:122] "RemoveContainer" containerID="51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84" Nov 03 00:31:48.793605 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.793586 4304 scope.go:122] "RemoveContainer" containerID="51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84" Nov 03 00:31:48.793877 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:48.793862 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84\": container with ID starting with 51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84 not found: ID does not exist" containerID="51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84" Nov 03 00:31:48.793935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.793893 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84"} err="failed to get container status \"51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84\": rpc error: code = NotFound desc = could not find container \"51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84\": container with ID starting with 51ad149b300044b135ad2e33d3a643635ee3cf958a6fb6951d5063ee77a7bd84 not found: ID does not exist" Nov 03 00:31:48.918751 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.918717 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94"] Nov 03 00:31:48.947526 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:48.947447 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4zf94"] Nov 03 00:31:49.047930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:49.047895 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76"] Nov 03 00:31:49.050950 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:49.050926 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6wc76"] Nov 03 00:31:49.114068 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:49.114031 4304 patch_prober.go:29] interesting pod/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:31:49.114245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:49.114096 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 00:31:49.149385 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:49.149350 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf"] Nov 03 00:31:49.152830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:49.152803 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8x2nf"] Nov 03 00:31:49.262050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:49.261942 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5"] Nov 03 00:31:49.267116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:49.267078 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r66p5"] Nov 03 00:31:50.113746 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:50.113714 4304 patch_prober.go:29] interesting pod/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:31:50.114347 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:50.113795 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 00:31:50.854989 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:50.854958 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" path="/var/lib/kubelet/pods/054e4951-35e0-44e9-a75d-a6d4cb457b1d/volumes" Nov 03 00:31:50.855428 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:50.855415 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" path="/var/lib/kubelet/pods/4dd0d831-a1b5-4ff0-ab36-726b1c1762ae/volumes" Nov 03 00:31:50.855822 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:50.855809 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" path="/var/lib/kubelet/pods/cb8636cf-f4f7-440d-86a4-fb4dc5aa771c/volumes" Nov 03 00:31:50.856519 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:50.856504 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" path="/var/lib/kubelet/pods/f456d8be-bb77-48f5-83fa-f684bcbc6cb9/volumes" Nov 03 00:31:51.113827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:51.113722 4304 patch_prober.go:29] interesting pod/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:31:51.114221 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:51.113816 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 00:31:52.113761 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.113728 4304 patch_prober.go:29] interesting pod/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:31:52.113969 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.113800 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 00:31:52.150054 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150019 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj"] Nov 03 00:31:52.150536 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150522 4304 cpu_manager.go:414] "containerMap: removing container" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerName="istio-proxy" Nov 03 00:31:52.150578 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150538 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerName="istio-proxy" Nov 03 00:31:52.150578 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150553 4304 cpu_manager.go:414] "containerMap: removing container" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" Nov 03 00:31:52.150578 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150557 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" Nov 03 00:31:52.150578 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150564 4304 cpu_manager.go:414] "containerMap: removing container" podUID="b9315860-a8bf-4d57-b752-2170be50dd40" containerName="payload-processing" Nov 03 00:31:52.150578 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150569 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9315860-a8bf-4d57-b752-2170be50dd40" containerName="payload-processing" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150587 4304 cpu_manager.go:414] "containerMap: removing container" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150592 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150601 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150606 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150611 4304 cpu_manager.go:414] "containerMap: removing container" podUID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" containerName="istio-proxy" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150616 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" containerName="istio-proxy" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150685 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd0d831-a1b5-4ff0-ab36-726b1c1762ae" containerName="istio-proxy" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150693 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="054e4951-35e0-44e9-a75d-a6d4cb457b1d" containerName="istio-proxy" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150699 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9315860-a8bf-4d57-b752-2170be50dd40" containerName="payload-processing" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150707 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8636cf-f4f7-440d-86a4-fb4dc5aa771c" containerName="istio-proxy" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150713 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f456d8be-bb77-48f5-83fa-f684bcbc6cb9" containerName="istio-proxy" Nov 03 00:31:52.150727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.150718 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2cf84f-1a22-49da-80a3-56ce09040f1c" containerName="istio-proxy" Nov 03 00:31:52.155731 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.155708 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:31:52.199922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.199898 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj"] Nov 03 00:31:52.217097 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.217068 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-38f5a3\"" Nov 03 00:31:52.231802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.220811 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lf7\" (UniqueName: \"kubernetes.io/projected/d539e6a6-1183-4598-908d-8a356f55f3d0-kube-api-access-z5lf7\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj\" (UID: \"d539e6a6-1183-4598-908d-8a356f55f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:31:52.231802 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.220879 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d539e6a6-1183-4598-908d-8a356f55f3d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj\" (UID: \"d539e6a6-1183-4598-908d-8a356f55f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:31:52.246084 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.246052 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-38f5a3-dockercfg-cw54n\"" Nov 03 00:31:52.346569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.346521 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lf7\" (UniqueName: \"kubernetes.io/projected/d539e6a6-1183-4598-908d-8a356f55f3d0-kube-api-access-z5lf7\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj\" (UID: \"d539e6a6-1183-4598-908d-8a356f55f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:31:52.346765 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.346578 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d539e6a6-1183-4598-908d-8a356f55f3d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj\" (UID: \"d539e6a6-1183-4598-908d-8a356f55f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:31:52.347252 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.347230 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d539e6a6-1183-4598-908d-8a356f55f3d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj\" (UID: \"d539e6a6-1183-4598-908d-8a356f55f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:31:52.403275 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.403242 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lf7\" (UniqueName: \"kubernetes.io/projected/d539e6a6-1183-4598-908d-8a356f55f3d0-kube-api-access-z5lf7\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj\" (UID: \"d539e6a6-1183-4598-908d-8a356f55f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:31:52.468992 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.468956 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:31:52.856080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.856006 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt"] Nov 03 00:31:52.861001 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.860983 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:31:52.871119 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:31:52.871098 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd539e6a6_1183_4598_908d_8a356f55f3d0.slice/crio-2522eb3fb1946c45a4658e81d2d684603490c052901baf8e3f758dca262496b7 WatchSource:0}: Error finding container 2522eb3fb1946c45a4658e81d2d684603490c052901baf8e3f758dca262496b7: Status 404 returned error can't find the container with id 2522eb3fb1946c45a4658e81d2d684603490c052901baf8e3f758dca262496b7 Nov 03 00:31:52.871266 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.871247 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj"] Nov 03 00:31:52.917768 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.917740 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt"] Nov 03 00:31:52.967801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.959952 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8769d72a-8592-4bed-867f-8ec05bdf8f15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt\" (UID: \"8769d72a-8592-4bed-867f-8ec05bdf8f15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:31:52.967801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:52.960046 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfrdj\" (UniqueName: \"kubernetes.io/projected/8769d72a-8592-4bed-867f-8ec05bdf8f15-kube-api-access-qfrdj\") pod \"payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt\" (UID: \"8769d72a-8592-4bed-867f-8ec05bdf8f15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:31:53.061277 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:53.061242 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8769d72a-8592-4bed-867f-8ec05bdf8f15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt\" (UID: \"8769d72a-8592-4bed-867f-8ec05bdf8f15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:31:53.061492 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:53.061307 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfrdj\" (UniqueName: \"kubernetes.io/projected/8769d72a-8592-4bed-867f-8ec05bdf8f15-kube-api-access-qfrdj\") pod \"payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt\" (UID: \"8769d72a-8592-4bed-867f-8ec05bdf8f15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:31:53.062189 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:53.062164 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8769d72a-8592-4bed-867f-8ec05bdf8f15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt\" (UID: \"8769d72a-8592-4bed-867f-8ec05bdf8f15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:31:53.139855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:53.127241 4304 patch_prober.go:29] interesting pod/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:31:53.139855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:53.127305 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 00:31:53.151875 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:53.146800 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfrdj\" (UniqueName: \"kubernetes.io/projected/8769d72a-8592-4bed-867f-8ec05bdf8f15-kube-api-access-qfrdj\") pod \"payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt\" (UID: \"8769d72a-8592-4bed-867f-8ec05bdf8f15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:31:53.188181 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:53.186346 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:31:53.573666 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:53.573636 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt"] Nov 03 00:31:53.574900 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:31:53.574873 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8769d72a_8592_4bed_867f_8ec05bdf8f15.slice/crio-d88aca9683b6b4709ee28fc3c6903e8fe041cb47cfba6fd80721eb9a2cf468dd WatchSource:0}: Error finding container d88aca9683b6b4709ee28fc3c6903e8fe041cb47cfba6fd80721eb9a2cf468dd: Status 404 returned error can't find the container with id d88aca9683b6b4709ee28fc3c6903e8fe041cb47cfba6fd80721eb9a2cf468dd Nov 03 00:31:53.807392 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:53.807357 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" event={"ID":"8769d72a-8592-4bed-867f-8ec05bdf8f15","Type":"ContainerStarted","Data":"d88aca9683b6b4709ee28fc3c6903e8fe041cb47cfba6fd80721eb9a2cf468dd"} Nov 03 00:31:53.808940 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:53.808904 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" event={"ID":"d539e6a6-1183-4598-908d-8a356f55f3d0","Type":"ContainerStarted","Data":"2522eb3fb1946c45a4658e81d2d684603490c052901baf8e3f758dca262496b7"} Nov 03 00:31:53.963513 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:53.963468 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:31:54.120413 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:54.120382 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:31:54.820815 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:54.820730 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:31:55.828651 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:55.828558 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" event={"ID":"8769d72a-8592-4bed-867f-8ec05bdf8f15","Type":"ContainerStarted","Data":"2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e"} Nov 03 00:31:55.828651 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:55.828638 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:31:55.830828 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:55.830804 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" event={"ID":"d539e6a6-1183-4598-908d-8a356f55f3d0","Type":"ContainerStarted","Data":"d229f1459ba4deae15067e149f61d8e5c94c7fa3be7c1756525371dbc7cd0cc9"} Nov 03 00:31:55.830972 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:55.830891 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:31:55.944292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:55.944233 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" podStartSLOduration=2.069600662 podStartE2EDuration="3.944216719s" totalImagesPullingTime="1.874616057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:55.9417506 +0000 UTC m=+1599.225308275" watchObservedRunningTime="2026-11-03 00:31:55.944216719 +0000 UTC m=+1599.227774424" Nov 03 00:31:57.224798 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:57.224710 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" podStartSLOduration=2.657352343 podStartE2EDuration="5.224690623s" totalImagesPullingTime="2.56733828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:56.059373446 +0000 UTC m=+1599.342931102" watchObservedRunningTime="2026-11-03 00:31:57.224690623 +0000 UTC m=+1600.508248299" Nov 03 00:31:57.227688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:31:57.227666 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt"] Nov 03 00:31:58.655541 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:31:58.655500 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:32:02.228656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.228618 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" podUID="589736ae-24c5-4a6a-8adc-039de3029fe0" containerName="payload-processing" containerID="cri-o://e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91" gracePeriod=25 Nov 03 00:32:02.497885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.497812 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:32:02.589302 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.589269 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589736ae-24c5-4a6a-8adc-039de3029fe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589736ae-24c5-4a6a-8adc-039de3029fe0-plugins-config-volume\") pod \"589736ae-24c5-4a6a-8adc-039de3029fe0\" (UID: \"589736ae-24c5-4a6a-8adc-039de3029fe0\") " Nov 03 00:32:02.589502 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.589422 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589736ae-24c5-4a6a-8adc-039de3029fe0-kube-api-access-5tctm\" (UniqueName: \"kubernetes.io/projected/589736ae-24c5-4a6a-8adc-039de3029fe0-kube-api-access-5tctm\") pod \"589736ae-24c5-4a6a-8adc-039de3029fe0\" (UID: \"589736ae-24c5-4a6a-8adc-039de3029fe0\") " Nov 03 00:32:02.589745 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.589714 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589736ae-24c5-4a6a-8adc-039de3029fe0-plugins-config-volume" pod "589736ae-24c5-4a6a-8adc-039de3029fe0" (UID: "589736ae-24c5-4a6a-8adc-039de3029fe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:32:02.590289 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.590270 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589736ae-24c5-4a6a-8adc-039de3029fe0-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:02.591788 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.591754 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589736ae-24c5-4a6a-8adc-039de3029fe0-kube-api-access-5tctm" pod "589736ae-24c5-4a6a-8adc-039de3029fe0" (UID: "589736ae-24c5-4a6a-8adc-039de3029fe0"). InnerVolumeSpecName "kube-api-access-5tctm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:32:02.691795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.691735 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tctm\" (UniqueName: \"kubernetes.io/projected/589736ae-24c5-4a6a-8adc-039de3029fe0-kube-api-access-5tctm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:02.877813 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.877761 4304 generic.go:356] "Generic (PLEG): container finished" podID="589736ae-24c5-4a6a-8adc-039de3029fe0" containerID="e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91" exitCode=0 Nov 03 00:32:02.877958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.877825 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" Nov 03 00:32:02.877958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.877846 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" event={"ID":"589736ae-24c5-4a6a-8adc-039de3029fe0","Type":"ContainerDied","Data":"e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91"} Nov 03 00:32:02.877958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.877901 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt" event={"ID":"589736ae-24c5-4a6a-8adc-039de3029fe0","Type":"ContainerDied","Data":"5bb49c3ff32e98814a2b46a679b1854d4e33d53b76447f563bbcb4ae64e055bd"} Nov 03 00:32:02.877958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.877929 4304 scope.go:122] "RemoveContainer" containerID="e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91" Nov 03 00:32:02.887902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.887881 4304 scope.go:122] "RemoveContainer" containerID="e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91" Nov 03 00:32:02.888220 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:32:02.888187 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91\": container with ID starting with e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91 not found: ID does not exist" containerID="e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91" Nov 03 00:32:02.888328 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:02.888223 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91"} err="failed to get container status \"e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91\": rpc error: code = NotFound desc = could not find container \"e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91\": container with ID starting with e46e8ebe79fddf18b7dc048dab98418642b0b137865bd857b3416177aa800e91 not found: ID does not exist" Nov 03 00:32:03.044902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.044873 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt"] Nov 03 00:32:03.049966 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.049937 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c521ba-86875995bf-9mltt"] Nov 03 00:32:03.527346 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.527306 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282"] Nov 03 00:32:03.527995 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.527976 4304 cpu_manager.go:414] "containerMap: removing container" podUID="589736ae-24c5-4a6a-8adc-039de3029fe0" containerName="payload-processing" Nov 03 00:32:03.528042 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.527996 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589736ae-24c5-4a6a-8adc-039de3029fe0" containerName="payload-processing" Nov 03 00:32:03.528091 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.528082 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="589736ae-24c5-4a6a-8adc-039de3029fe0" containerName="payload-processing" Nov 03 00:32:03.532570 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.532554 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.584476 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.584454 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8c9581-serving-cert\"" Nov 03 00:32:03.592080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.592054 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282"] Nov 03 00:32:03.702377 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.702344 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.702544 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.702482 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-metrics-certs\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.702589 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.702572 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.702625 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.702604 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqrf\" (UniqueName: \"kubernetes.io/projected/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-kube-api-access-mpqrf\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.804199 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.804118 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-metrics-certs\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.804199 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.804178 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.804396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.804206 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqrf\" (UniqueName: \"kubernetes.io/projected/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-kube-api-access-mpqrf\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.804396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.804279 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.804901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.804874 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.806699 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.806678 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.806838 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.806732 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-metrics-certs\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:03.860863 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:03.860835 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqrf\" (UniqueName: \"kubernetes.io/projected/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-kube-api-access-mpqrf\") pod \"maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:04.145420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:04.145393 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:04.519186 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:04.519163 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282"] Nov 03 00:32:04.520769 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:32:04.520737 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6fec01_18e7_4980_8cc2_7cf2344b0f6f.slice/crio-9df14bf68f062811b316e43bb0ee38fa9de0f19ce125b71b116dcf55e0bbe358 WatchSource:0}: Error finding container 9df14bf68f062811b316e43bb0ee38fa9de0f19ce125b71b116dcf55e0bbe358: Status 404 returned error can't find the container with id 9df14bf68f062811b316e43bb0ee38fa9de0f19ce125b71b116dcf55e0bbe358 Nov 03 00:32:04.829738 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:04.829667 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589736ae-24c5-4a6a-8adc-039de3029fe0" path="/var/lib/kubelet/pods/589736ae-24c5-4a6a-8adc-039de3029fe0/volumes" Nov 03 00:32:04.896499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:04.896464 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" event={"ID":"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f","Type":"ContainerStarted","Data":"9df14bf68f062811b316e43bb0ee38fa9de0f19ce125b71b116dcf55e0bbe358"} Nov 03 00:32:05.903864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:05.903826 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" event={"ID":"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f","Type":"ContainerStarted","Data":"8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82"} Nov 03 00:32:05.904276 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:05.903907 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:06.018437 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:06.018385 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" podStartSLOduration=2.193312388 podStartE2EDuration="3.018365755s" totalImagesPullingTime="825.053367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:06.017851938 +0000 UTC m=+1609.301409612" watchObservedRunningTime="2026-11-03 00:32:06.018365755 +0000 UTC m=+1609.301923430" Nov 03 00:32:06.838887 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:06.838859 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:32:06.839089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:06.838930 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:32:11.009676 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.009643 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd"] Nov 03 00:32:11.013763 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.013743 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.065449 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.065425 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cb65f3a5-openshift-default-dockercfg-fxcdx\"" Nov 03 00:32:11.087445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.087409 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.087591 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.087481 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.087591 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.087552 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m48w\" (UniqueName: \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-kube-api-access-4m48w\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.087687 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.087612 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.087729 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.087666 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.087813 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.087795 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.087885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.087869 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8137f3ca-7b7e-40e4-95e0-c97518557954-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.087924 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.087913 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.087998 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.087980 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.118656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.118621 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd"] Nov 03 00:32:11.188720 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.188678 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.188937 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.188787 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.188937 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.188827 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m48w\" (UniqueName: \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-kube-api-access-4m48w\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.188937 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.188893 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.188937 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.188922 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.189177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.188946 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.189177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.189012 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8137f3ca-7b7e-40e4-95e0-c97518557954-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.189177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.189049 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.189177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.189080 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.189370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.189227 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.189444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.189420 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.189671 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.189481 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.189671 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.189499 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.189813 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.189719 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8137f3ca-7b7e-40e4-95e0-c97518557954-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.191258 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.191230 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.191629 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.191606 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.245575 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.245532 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.245744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.245628 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m48w\" (UniqueName: \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-kube-api-access-4m48w\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.327202 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.327110 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:11.753444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.753040 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd"] Nov 03 00:32:11.759396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.759360 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:11.759533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.759439 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:11.759533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.759485 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:11.915723 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.915696 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:32:11.942726 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.942684 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" event={"ID":"8137f3ca-7b7e-40e4-95e0-c97518557954","Type":"ContainerStarted","Data":"366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7"} Nov 03 00:32:11.942898 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:11.942748 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" event={"ID":"8137f3ca-7b7e-40e4-95e0-c97518557954","Type":"ContainerStarted","Data":"5e85a44f1b8ce75c10a7c68caae60f85336167190b46e86637c81a0fe15f28a7"} Nov 03 00:32:12.149422 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:12.149359 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" podStartSLOduration=2.1493377049999998 podStartE2EDuration="2.149337705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:12.148173002 +0000 UTC m=+1615.431730694" watchObservedRunningTime="2026-11-03 00:32:12.149337705 +0000 UTC m=+1615.432895451" Nov 03 00:32:12.338645 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:12.338187 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:12.340292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:12.340258 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:32:12.340523 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:12.340495 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" podUID="8137f3ca-7b7e-40e4-95e0-c97518557954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 00:32:13.345803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:13.336143 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:32:13.346198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:13.345883 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" podUID="8137f3ca-7b7e-40e4-95e0-c97518557954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 00:32:14.341833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:14.341205 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:32:14.341833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:14.341267 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" podUID="8137f3ca-7b7e-40e4-95e0-c97518557954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 00:32:15.328863 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:15.328833 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:32:15.329420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:15.329389 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" podUID="8137f3ca-7b7e-40e4-95e0-c97518557954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 00:32:16.344799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:16.341179 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:32:16.344799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:16.341255 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" podUID="8137f3ca-7b7e-40e4-95e0-c97518557954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 00:32:17.333801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:17.329225 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:32:17.333801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:17.329305 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" podUID="8137f3ca-7b7e-40e4-95e0-c97518557954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 00:32:18.334053 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:18.334005 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:18.334714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:18.334688 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:18.335546 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:18.335511 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:32:26.146296 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.146260 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m"] Nov 03 00:32:26.151818 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.151801 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.308344 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.308308 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.308504 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.308364 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.308504 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.308473 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.308577 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.308516 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2wr\" (UniqueName: \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-kube-api-access-gf2wr\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.308616 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.308575 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.308650 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.308628 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.308686 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.308670 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.308840 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.308818 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.308881 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.308855 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68711b8-ca31-416b-950f-26b3d43f73ad-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.360442 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.360412 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m"] Nov 03 00:32:26.410136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410054 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.410136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410088 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68711b8-ca31-416b-950f-26b3d43f73ad-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.410361 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410143 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.410361 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410176 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.410361 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410201 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.410361 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410232 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2wr\" (UniqueName: \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-kube-api-access-gf2wr\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.410361 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410286 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.410361 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410350 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.410696 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410410 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.410696 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410583 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.410850 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410718 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.410850 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410832 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.411154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410867 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.411154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.410981 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68711b8-ca31-416b-950f-26b3d43f73ad-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.412815 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.412788 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.412925 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.412857 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.466818 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.466765 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2wr\" (UniqueName: \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-kube-api-access-gf2wr\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.467072 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.467056 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:26.766982 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:26.766901 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:27.195268 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:32:27.195239 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68711b8_ca31_416b_950f_26b3d43f73ad.slice/crio-99555f2310cbcc9c2f3f8ce83e2724c75a65376ecee300f7540dfce0bf994201 WatchSource:0}: Error finding container 99555f2310cbcc9c2f3f8ce83e2724c75a65376ecee300f7540dfce0bf994201: Status 404 returned error can't find the container with id 99555f2310cbcc9c2f3f8ce83e2724c75a65376ecee300f7540dfce0bf994201 Nov 03 00:32:27.195599 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:27.195449 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m"] Nov 03 00:32:27.197854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:27.197820 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:27.197959 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:27.197885 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:27.197959 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:27.197918 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:28.044627 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:28.044592 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" event={"ID":"d68711b8-ca31-416b-950f-26b3d43f73ad","Type":"ContainerStarted","Data":"0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4"} Nov 03 00:32:28.044876 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:28.044652 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" event={"ID":"d68711b8-ca31-416b-950f-26b3d43f73ad","Type":"ContainerStarted","Data":"99555f2310cbcc9c2f3f8ce83e2724c75a65376ecee300f7540dfce0bf994201"} Nov 03 00:32:28.201530 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:28.201474 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" podStartSLOduration=2.201458254 podStartE2EDuration="2.201458254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:28.198247128 +0000 UTC m=+1631.481804793" watchObservedRunningTime="2026-11-03 00:32:28.201458254 +0000 UTC m=+1631.485015928" Nov 03 00:32:28.676911 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:32:28.676875 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:32:28.768048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:28.768003 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:28.769859 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:28.769824 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:32:28.769975 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:28.769877 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" podUID="d68711b8-ca31-416b-950f-26b3d43f73ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 00:32:29.767884 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:29.767846 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:32:29.768302 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:29.767902 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" podUID="d68711b8-ca31-416b-950f-26b3d43f73ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 00:32:30.767658 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:30.767620 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:32:30.767871 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:30.767682 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" podUID="d68711b8-ca31-416b-950f-26b3d43f73ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 00:32:31.767543 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:31.767513 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:32:31.767972 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:31.767564 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" podUID="d68711b8-ca31-416b-950f-26b3d43f73ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 00:32:32.767657 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:32.767616 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:32:32.768086 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:32.767680 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" podUID="d68711b8-ca31-416b-950f-26b3d43f73ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 00:32:33.767396 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:33.767353 4304 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:32:33.767554 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:33.767420 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" podUID="d68711b8-ca31-416b-950f-26b3d43f73ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 00:32:34.772276 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:34.772244 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:34.772827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:34.772803 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:34.773806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:32:34.773759 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:32:58.654515 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:32:58.654474 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:33:02.849386 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:02.849301 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf"] Nov 03 00:33:02.849858 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:02.849599 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" containerID="cri-o://eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108" gracePeriod=30 Nov 03 00:33:08.141899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.141871 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:33:08.308380 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.308343 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/833edfe8-d264-4331-947f-6648bd594ff8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833edfe8-d264-4331-947f-6648bd594ff8-istio-podinfo\") pod \"833edfe8-d264-4331-947f-6648bd594ff8\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " Nov 03 00:33:08.308586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.308410 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-istio-token\" (UniqueName: \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-istio-token\") pod \"833edfe8-d264-4331-947f-6648bd594ff8\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " Nov 03 00:33:08.308586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.308445 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-socket\") pod \"833edfe8-d264-4331-947f-6648bd594ff8\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " Nov 03 00:33:08.308586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.308497 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-envoy\") pod \"833edfe8-d264-4331-947f-6648bd594ff8\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " Nov 03 00:33:08.308586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.308557 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-kube-api-access-288s2\" (UniqueName: \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-kube-api-access-288s2\") pod \"833edfe8-d264-4331-947f-6648bd594ff8\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " Nov 03 00:33:08.308842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.308605 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-certs\") pod \"833edfe8-d264-4331-947f-6648bd594ff8\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " Nov 03 00:33:08.308842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.308637 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-data\") pod \"833edfe8-d264-4331-947f-6648bd594ff8\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " Nov 03 00:33:08.308842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.308720 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/833edfe8-d264-4331-947f-6648bd594ff8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833edfe8-d264-4331-947f-6648bd594ff8-istiod-ca-cert\") pod \"833edfe8-d264-4331-947f-6648bd594ff8\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " Nov 03 00:33:08.308842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.308721 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-socket" pod "833edfe8-d264-4331-947f-6648bd594ff8" (UID: "833edfe8-d264-4331-947f-6648bd594ff8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:08.308842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.308760 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-credential-socket\") pod \"833edfe8-d264-4331-947f-6648bd594ff8\" (UID: \"833edfe8-d264-4331-947f-6648bd594ff8\") " Nov 03 00:33:08.309097 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.308913 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-certs" pod "833edfe8-d264-4331-947f-6648bd594ff8" (UID: "833edfe8-d264-4331-947f-6648bd594ff8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:08.309155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.309090 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-data" pod "833edfe8-d264-4331-947f-6648bd594ff8" (UID: "833edfe8-d264-4331-947f-6648bd594ff8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:08.309524 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.309495 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-credential-socket" pod "833edfe8-d264-4331-947f-6648bd594ff8" (UID: "833edfe8-d264-4331-947f-6648bd594ff8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:08.309636 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.309540 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:08.309636 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.309564 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:08.309636 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.309580 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:08.309822 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.309747 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833edfe8-d264-4331-947f-6648bd594ff8-istiod-ca-cert" pod "833edfe8-d264-4331-947f-6648bd594ff8" (UID: "833edfe8-d264-4331-947f-6648bd594ff8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:08.311203 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.311166 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/833edfe8-d264-4331-947f-6648bd594ff8-istio-podinfo" pod "833edfe8-d264-4331-947f-6648bd594ff8" (UID: "833edfe8-d264-4331-947f-6648bd594ff8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:33:08.311432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.311404 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-envoy" pod "833edfe8-d264-4331-947f-6648bd594ff8" (UID: "833edfe8-d264-4331-947f-6648bd594ff8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:08.312034 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.312005 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-istio-token" pod "833edfe8-d264-4331-947f-6648bd594ff8" (UID: "833edfe8-d264-4331-947f-6648bd594ff8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:08.312147 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.312075 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-kube-api-access-288s2" pod "833edfe8-d264-4331-947f-6648bd594ff8" (UID: "833edfe8-d264-4331-947f-6648bd594ff8"). InnerVolumeSpecName "kube-api-access-288s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:08.339879 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.339851 4304 generic.go:356] "Generic (PLEG): container finished" podID="833edfe8-d264-4331-947f-6648bd594ff8" containerID="eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108" exitCode=0 Nov 03 00:33:08.340009 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.339904 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" Nov 03 00:33:08.340009 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.339930 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" event={"ID":"833edfe8-d264-4331-947f-6648bd594ff8","Type":"ContainerDied","Data":"eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108"} Nov 03 00:33:08.340009 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.339989 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf" event={"ID":"833edfe8-d264-4331-947f-6648bd594ff8","Type":"ContainerDied","Data":"1f79faeb902da9f6f0c71e922d1c0f341cfc76c55168cdab57c87570a05dc6d6"} Nov 03 00:33:08.340116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.340019 4304 scope.go:122] "RemoveContainer" containerID="eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108" Nov 03 00:33:08.350949 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.350926 4304 scope.go:122] "RemoveContainer" containerID="9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36" Nov 03 00:33:08.362076 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.362054 4304 scope.go:122] "RemoveContainer" containerID="eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108" Nov 03 00:33:08.362383 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:33:08.362361 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108\": container with ID starting with eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108 not found: ID does not exist" containerID="eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108" Nov 03 00:33:08.362435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.362392 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108"} err="failed to get container status \"eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108\": rpc error: code = NotFound desc = could not find container \"eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108\": container with ID starting with eb73940033f1a93b17a5c542a28865b1911fd7fa83350a5131bbeed092cb4108 not found: ID does not exist" Nov 03 00:33:08.362435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.362409 4304 scope.go:122] "RemoveContainer" containerID="9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36" Nov 03 00:33:08.362681 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:33:08.362651 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36\": container with ID starting with 9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36 not found: ID does not exist" containerID="9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36" Nov 03 00:33:08.362736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.362692 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36"} err="failed to get container status \"9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36\": rpc error: code = NotFound desc = could not find container \"9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36\": container with ID starting with 9139b300838d57637073aa52541cade20fe1cb24c1c58a621c461eca9d3a8f36 not found: ID does not exist" Nov 03 00:33:08.410163 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.410139 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833edfe8-d264-4331-947f-6648bd594ff8-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:08.410163 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.410162 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:08.410362 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.410172 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833edfe8-d264-4331-947f-6648bd594ff8-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:08.410362 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.410184 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:08.410362 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.410193 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833edfe8-d264-4331-947f-6648bd594ff8-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:08.410362 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.410203 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-288s2\" (UniqueName: \"kubernetes.io/projected/833edfe8-d264-4331-947f-6648bd594ff8-kube-api-access-288s2\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:08.519310 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.519280 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf"] Nov 03 00:33:08.524856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.524829 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c521ba-openshift-default-646df59755-75nrf"] Nov 03 00:33:08.830747 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:08.830674 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833edfe8-d264-4331-947f-6648bd594ff8" path="/var/lib/kubelet/pods/833edfe8-d264-4331-947f-6648bd594ff8/volumes" Nov 03 00:33:14.290368 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.290335 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc"] Nov 03 00:33:14.290921 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.290894 4304 cpu_manager.go:414] "containerMap: removing container" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" Nov 03 00:33:14.290921 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.290912 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" Nov 03 00:33:14.291105 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.290950 4304 cpu_manager.go:414] "containerMap: removing container" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" Nov 03 00:33:14.291105 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.290957 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" Nov 03 00:33:14.291216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.291105 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" Nov 03 00:33:14.291216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.291122 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="833edfe8-d264-4331-947f-6648bd594ff8" containerName="istio-proxy" Nov 03 00:33:14.294699 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.294681 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:33:14.363177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.363144 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc"] Nov 03 00:33:14.405717 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.405669 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4qlj\" (UniqueName: \"kubernetes.io/projected/903d7c7f-1e87-4385-9548-638ff9390eae-kube-api-access-c4qlj\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc\" (UID: \"903d7c7f-1e87-4385-9548-638ff9390eae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:33:14.406350 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.406312 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d7c7f-1e87-4385-9548-638ff9390eae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc\" (UID: \"903d7c7f-1e87-4385-9548-638ff9390eae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:33:14.409457 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.409438 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1a883c\"" Nov 03 00:33:14.409711 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.409697 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1a883c-dockercfg-vnvrr\"" Nov 03 00:33:14.468246 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.468208 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6"] Nov 03 00:33:14.473457 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.473428 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:33:14.507342 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.507313 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d7c7f-1e87-4385-9548-638ff9390eae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc\" (UID: \"903d7c7f-1e87-4385-9548-638ff9390eae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:33:14.507491 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.507385 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4qlj\" (UniqueName: \"kubernetes.io/projected/903d7c7f-1e87-4385-9548-638ff9390eae-kube-api-access-c4qlj\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc\" (UID: \"903d7c7f-1e87-4385-9548-638ff9390eae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:33:14.507962 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.507942 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d7c7f-1e87-4385-9548-638ff9390eae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc\" (UID: \"903d7c7f-1e87-4385-9548-638ff9390eae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:33:14.546701 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.546636 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6"] Nov 03 00:33:14.565412 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.565383 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4qlj\" (UniqueName: \"kubernetes.io/projected/903d7c7f-1e87-4385-9548-638ff9390eae-kube-api-access-c4qlj\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc\" (UID: \"903d7c7f-1e87-4385-9548-638ff9390eae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:33:14.607827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.607796 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:33:14.608144 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.608127 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd605632-b5ae-4448-88fc-eb3638a4244b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6\" (UID: \"fd605632-b5ae-4448-88fc-eb3638a4244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:33:14.608328 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.608313 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68fd\" (UniqueName: \"kubernetes.io/projected/fd605632-b5ae-4448-88fc-eb3638a4244b-kube-api-access-z68fd\") pod \"payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6\" (UID: \"fd605632-b5ae-4448-88fc-eb3638a4244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:33:14.710169 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.710124 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68fd\" (UniqueName: \"kubernetes.io/projected/fd605632-b5ae-4448-88fc-eb3638a4244b-kube-api-access-z68fd\") pod \"payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6\" (UID: \"fd605632-b5ae-4448-88fc-eb3638a4244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:33:14.710591 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.710559 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd605632-b5ae-4448-88fc-eb3638a4244b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6\" (UID: \"fd605632-b5ae-4448-88fc-eb3638a4244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:33:14.711213 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.711195 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd605632-b5ae-4448-88fc-eb3638a4244b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6\" (UID: \"fd605632-b5ae-4448-88fc-eb3638a4244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:33:14.767703 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.767675 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68fd\" (UniqueName: \"kubernetes.io/projected/fd605632-b5ae-4448-88fc-eb3638a4244b-kube-api-access-z68fd\") pod \"payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6\" (UID: \"fd605632-b5ae-4448-88fc-eb3638a4244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:33:14.787914 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.787890 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:33:14.989683 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:14.989660 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc"] Nov 03 00:33:14.991933 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:33:14.991907 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903d7c7f_1e87_4385_9548_638ff9390eae.slice/crio-2cf3ecb37f0b02ca63f740c3243cd2ce1055e4309be78bed7fc6a04ddd9359be WatchSource:0}: Error finding container 2cf3ecb37f0b02ca63f740c3243cd2ce1055e4309be78bed7fc6a04ddd9359be: Status 404 returned error can't find the container with id 2cf3ecb37f0b02ca63f740c3243cd2ce1055e4309be78bed7fc6a04ddd9359be Nov 03 00:33:15.163710 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:15.163678 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6"] Nov 03 00:33:15.165281 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:33:15.165259 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd605632_b5ae_4448_88fc_eb3638a4244b.slice/crio-4245278b471a7f4460f875798b815adbdea5a6d454ceb8dfa4afbe46c1099668 WatchSource:0}: Error finding container 4245278b471a7f4460f875798b815adbdea5a6d454ceb8dfa4afbe46c1099668: Status 404 returned error can't find the container with id 4245278b471a7f4460f875798b815adbdea5a6d454ceb8dfa4afbe46c1099668 Nov 03 00:33:15.411969 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:15.411869 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" event={"ID":"903d7c7f-1e87-4385-9548-638ff9390eae","Type":"ContainerStarted","Data":"56cea79f85b9532908d8217a0edc42e65af567d07cfaeee592e76a3899e03b07"} Nov 03 00:33:15.411969 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:15.411934 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" event={"ID":"903d7c7f-1e87-4385-9548-638ff9390eae","Type":"ContainerStarted","Data":"2cf3ecb37f0b02ca63f740c3243cd2ce1055e4309be78bed7fc6a04ddd9359be"} Nov 03 00:33:15.412503 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:15.412117 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:33:15.413864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:15.413839 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" event={"ID":"fd605632-b5ae-4448-88fc-eb3638a4244b","Type":"ContainerStarted","Data":"1aafcf855384b98cfbbd6526823ed5bf182b2adb963ab7f5deb2d14338fdf85b"} Nov 03 00:33:15.413989 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:15.413876 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" event={"ID":"fd605632-b5ae-4448-88fc-eb3638a4244b","Type":"ContainerStarted","Data":"4245278b471a7f4460f875798b815adbdea5a6d454ceb8dfa4afbe46c1099668"} Nov 03 00:33:15.413989 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:15.413969 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:33:15.530867 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:15.530814 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" podStartSLOduration=1.530800154 podStartE2EDuration="1.530800154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:15.527689173 +0000 UTC m=+1678.811246844" watchObservedRunningTime="2026-11-03 00:33:15.530800154 +0000 UTC m=+1678.814357827" Nov 03 00:33:15.644794 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:15.644718 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" podStartSLOduration=1.644704952 podStartE2EDuration="1.644704952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:15.64200549 +0000 UTC m=+1678.925563167" watchObservedRunningTime="2026-11-03 00:33:15.644704952 +0000 UTC m=+1678.928262665" Nov 03 00:33:17.543933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:17.543839 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w"] Nov 03 00:33:21.725559 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:21.725526 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8"] Nov 03 00:33:21.730833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:21.730815 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:33:21.791231 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:21.791195 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25334bc9-eb19-4379-a2f0-510f579bdeeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8\" (UID: \"25334bc9-eb19-4379-a2f0-510f579bdeeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:33:21.791401 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:21.791254 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h97sj\" (UniqueName: \"kubernetes.io/projected/25334bc9-eb19-4379-a2f0-510f579bdeeb-kube-api-access-h97sj\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8\" (UID: \"25334bc9-eb19-4379-a2f0-510f579bdeeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:33:21.833914 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:21.833889 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bb5643-dockercfg-fbqbr\"" Nov 03 00:33:21.834081 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:21.834044 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bb5643\"" Nov 03 00:33:21.837249 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:21.837228 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8"] Nov 03 00:33:21.892684 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:21.892656 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25334bc9-eb19-4379-a2f0-510f579bdeeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8\" (UID: \"25334bc9-eb19-4379-a2f0-510f579bdeeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:33:21.892878 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:21.892696 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h97sj\" (UniqueName: \"kubernetes.io/projected/25334bc9-eb19-4379-a2f0-510f579bdeeb-kube-api-access-h97sj\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8\" (UID: \"25334bc9-eb19-4379-a2f0-510f579bdeeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:33:21.895292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:21.894272 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25334bc9-eb19-4379-a2f0-510f579bdeeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8\" (UID: \"25334bc9-eb19-4379-a2f0-510f579bdeeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:33:21.949843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:21.949804 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h97sj\" (UniqueName: \"kubernetes.io/projected/25334bc9-eb19-4379-a2f0-510f579bdeeb-kube-api-access-h97sj\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8\" (UID: \"25334bc9-eb19-4379-a2f0-510f579bdeeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:33:22.044311 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.044239 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:33:22.419111 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.419083 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8"] Nov 03 00:33:22.420557 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:33:22.420526 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25334bc9_eb19_4379_a2f0_510f579bdeeb.slice/crio-15e822a23178defabe0e88917f0130f5536844d4334c6ce32259b670964e9f23 WatchSource:0}: Error finding container 15e822a23178defabe0e88917f0130f5536844d4334c6ce32259b670964e9f23: Status 404 returned error can't find the container with id 15e822a23178defabe0e88917f0130f5536844d4334c6ce32259b670964e9f23 Nov 03 00:33:22.463632 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.463589 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" event={"ID":"25334bc9-eb19-4379-a2f0-510f579bdeeb","Type":"ContainerStarted","Data":"15e822a23178defabe0e88917f0130f5536844d4334c6ce32259b670964e9f23"} Nov 03 00:33:22.530703 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.530667 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5"] Nov 03 00:33:22.534757 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.534735 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:33:22.545022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.544962 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" podUID="0622afbb-ae90-40fe-b918-910e6a035f85" containerName="payload-processing" containerID="cri-o://92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4" gracePeriod=25 Nov 03 00:33:22.593321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.593293 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5"] Nov 03 00:33:22.601520 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.601491 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6dtg\" (UniqueName: \"kubernetes.io/projected/e2551088-6824-4725-bc0f-c66120219137-kube-api-access-b6dtg\") pod \"payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5\" (UID: \"e2551088-6824-4725-bc0f-c66120219137\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:33:22.601721 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.601696 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2551088-6824-4725-bc0f-c66120219137-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5\" (UID: \"e2551088-6824-4725-bc0f-c66120219137\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:33:22.702690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.702593 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2551088-6824-4725-bc0f-c66120219137-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5\" (UID: \"e2551088-6824-4725-bc0f-c66120219137\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:33:22.702889 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.702729 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6dtg\" (UniqueName: \"kubernetes.io/projected/e2551088-6824-4725-bc0f-c66120219137-kube-api-access-b6dtg\") pod \"payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5\" (UID: \"e2551088-6824-4725-bc0f-c66120219137\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:33:22.703308 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.703284 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2551088-6824-4725-bc0f-c66120219137-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5\" (UID: \"e2551088-6824-4725-bc0f-c66120219137\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:33:22.759702 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.759667 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6dtg\" (UniqueName: \"kubernetes.io/projected/e2551088-6824-4725-bc0f-c66120219137-kube-api-access-b6dtg\") pod \"payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5\" (UID: \"e2551088-6824-4725-bc0f-c66120219137\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:33:22.791048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.791025 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:33:22.848763 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.848733 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:33:22.904871 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.904826 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0622afbb-ae90-40fe-b918-910e6a035f85-kube-api-access-9zjmc\" (UniqueName: \"kubernetes.io/projected/0622afbb-ae90-40fe-b918-910e6a035f85-kube-api-access-9zjmc\") pod \"0622afbb-ae90-40fe-b918-910e6a035f85\" (UID: \"0622afbb-ae90-40fe-b918-910e6a035f85\") " Nov 03 00:33:22.905305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.905031 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0622afbb-ae90-40fe-b918-910e6a035f85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0622afbb-ae90-40fe-b918-910e6a035f85-plugins-config-volume\") pod \"0622afbb-ae90-40fe-b918-910e6a035f85\" (UID: \"0622afbb-ae90-40fe-b918-910e6a035f85\") " Nov 03 00:33:22.920815 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.906952 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0622afbb-ae90-40fe-b918-910e6a035f85-plugins-config-volume" pod "0622afbb-ae90-40fe-b918-910e6a035f85" (UID: "0622afbb-ae90-40fe-b918-910e6a035f85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:22.920815 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:22.907483 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0622afbb-ae90-40fe-b918-910e6a035f85-kube-api-access-9zjmc" pod "0622afbb-ae90-40fe-b918-910e6a035f85" (UID: "0622afbb-ae90-40fe-b918-910e6a035f85"). InnerVolumeSpecName "kube-api-access-9zjmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:23.007022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.006991 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0622afbb-ae90-40fe-b918-910e6a035f85-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:23.007022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.007022 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zjmc\" (UniqueName: \"kubernetes.io/projected/0622afbb-ae90-40fe-b918-910e6a035f85-kube-api-access-9zjmc\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:23.221958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.221927 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5"] Nov 03 00:33:23.223065 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:33:23.223039 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2551088_6824_4725_bc0f_c66120219137.slice/crio-c84d01bfa3f48691de067bedd0dddd72353a04fac1d7bcd7f0481f2f4a5fdffe WatchSource:0}: Error finding container c84d01bfa3f48691de067bedd0dddd72353a04fac1d7bcd7f0481f2f4a5fdffe: Status 404 returned error can't find the container with id c84d01bfa3f48691de067bedd0dddd72353a04fac1d7bcd7f0481f2f4a5fdffe Nov 03 00:33:23.471441 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.471340 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" event={"ID":"25334bc9-eb19-4379-a2f0-510f579bdeeb","Type":"ContainerStarted","Data":"c30a95ba8d6801e085f081e51df8e256d9e90d1ce56e73a9afe4db8ac4c49e87"} Nov 03 00:33:23.471621 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.471548 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:33:23.472620 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.472597 4304 generic.go:356] "Generic (PLEG): container finished" podID="0622afbb-ae90-40fe-b918-910e6a035f85" containerID="92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4" exitCode=0 Nov 03 00:33:23.472761 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.472642 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" Nov 03 00:33:23.472761 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.472667 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" event={"ID":"0622afbb-ae90-40fe-b918-910e6a035f85","Type":"ContainerDied","Data":"92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4"} Nov 03 00:33:23.472761 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.472710 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w" event={"ID":"0622afbb-ae90-40fe-b918-910e6a035f85","Type":"ContainerDied","Data":"c37431a17b89ba52cc684881eae09b04d3a69e7f0ad6fd72fda0a74035cbdb6c"} Nov 03 00:33:23.472761 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.472731 4304 scope.go:122] "RemoveContainer" containerID="92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4" Nov 03 00:33:23.474410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.474388 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" event={"ID":"e2551088-6824-4725-bc0f-c66120219137","Type":"ContainerStarted","Data":"e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b"} Nov 03 00:33:23.474503 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.474440 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:33:23.474503 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.474467 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" event={"ID":"e2551088-6824-4725-bc0f-c66120219137","Type":"ContainerStarted","Data":"c84d01bfa3f48691de067bedd0dddd72353a04fac1d7bcd7f0481f2f4a5fdffe"} Nov 03 00:33:23.482567 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.482543 4304 scope.go:122] "RemoveContainer" containerID="92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4" Nov 03 00:33:23.482867 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:33:23.482849 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4\": container with ID starting with 92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4 not found: ID does not exist" containerID="92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4" Nov 03 00:33:23.482923 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.482875 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4"} err="failed to get container status \"92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4\": rpc error: code = NotFound desc = could not find container \"92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4\": container with ID starting with 92a5c1a42500a711672e15a177fa8d78157b07ee46b108de7d276f22df5f26e4 not found: ID does not exist" Nov 03 00:33:23.595331 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.595262 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" podStartSLOduration=2.595243604 podStartE2EDuration="2.595243604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:23.593144906 +0000 UTC m=+1686.876702581" watchObservedRunningTime="2026-11-03 00:33:23.595243604 +0000 UTC m=+1686.878801279" Nov 03 00:33:23.712529 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.712477 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" podStartSLOduration=1.712463964 podStartE2EDuration="1.712463964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:23.709158583 +0000 UTC m=+1686.992716259" watchObservedRunningTime="2026-11-03 00:33:23.712463964 +0000 UTC m=+1686.996021637" Nov 03 00:33:23.823213 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.823130 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w"] Nov 03 00:33:23.839218 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:23.839178 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-774876f949-94z5w"] Nov 03 00:33:24.830381 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:24.830346 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0622afbb-ae90-40fe-b918-910e6a035f85" path="/var/lib/kubelet/pods/0622afbb-ae90-40fe-b918-910e6a035f85/volumes" Nov 03 00:33:26.424736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:26.424698 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:33:26.425250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:26.424800 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:33:28.656471 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:33:28.656434 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:33:34.484835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:34.484799 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:33:34.485343 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:34.484980 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:33:47.323932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.323894 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt"] Nov 03 00:33:47.324437 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.324108 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" podUID="8769d72a-8592-4bed-867f-8ec05bdf8f15" containerName="payload-processing" containerID="cri-o://2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e" gracePeriod=30 Nov 03 00:33:47.430005 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.429971 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj"] Nov 03 00:33:47.430272 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.430227 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" podUID="d539e6a6-1183-4598-908d-8a356f55f3d0" containerName="payload-pre-processing" containerID="cri-o://d229f1459ba4deae15067e149f61d8e5c94c7fa3be7c1756525371dbc7cd0cc9" gracePeriod=30 Nov 03 00:33:47.627134 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.626789 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:33:47.631218 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.630792 4304 generic.go:356] "Generic (PLEG): container finished" podID="8769d72a-8592-4bed-867f-8ec05bdf8f15" containerID="2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e" exitCode=0 Nov 03 00:33:47.631218 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.630845 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" Nov 03 00:33:47.631218 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.630971 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" event={"ID":"8769d72a-8592-4bed-867f-8ec05bdf8f15","Type":"ContainerDied","Data":"2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e"} Nov 03 00:33:47.631218 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.631011 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt" event={"ID":"8769d72a-8592-4bed-867f-8ec05bdf8f15","Type":"ContainerDied","Data":"d88aca9683b6b4709ee28fc3c6903e8fe041cb47cfba6fd80721eb9a2cf468dd"} Nov 03 00:33:47.631218 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.631031 4304 scope.go:122] "RemoveContainer" containerID="2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e" Nov 03 00:33:47.633709 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.633547 4304 generic.go:356] "Generic (PLEG): container finished" podID="d539e6a6-1183-4598-908d-8a356f55f3d0" containerID="d229f1459ba4deae15067e149f61d8e5c94c7fa3be7c1756525371dbc7cd0cc9" exitCode=0 Nov 03 00:33:47.633709 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.633666 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" event={"ID":"d539e6a6-1183-4598-908d-8a356f55f3d0","Type":"ContainerDied","Data":"d229f1459ba4deae15067e149f61d8e5c94c7fa3be7c1756525371dbc7cd0cc9"} Nov 03 00:33:47.650131 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.650097 4304 scope.go:122] "RemoveContainer" containerID="2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e" Nov 03 00:33:47.650794 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:33:47.650757 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e\": container with ID starting with 2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e not found: ID does not exist" containerID="2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e" Nov 03 00:33:47.650931 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.650908 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e"} err="failed to get container status \"2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e\": rpc error: code = NotFound desc = could not find container \"2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e\": container with ID starting with 2be89d65c90f75bbf927842a84636e2415b117a25b7af9e106d0b2352a61b30e not found: ID does not exist" Nov 03 00:33:47.701933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.701887 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8769d72a-8592-4bed-867f-8ec05bdf8f15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8769d72a-8592-4bed-867f-8ec05bdf8f15-plugins-config-volume\") pod \"8769d72a-8592-4bed-867f-8ec05bdf8f15\" (UID: \"8769d72a-8592-4bed-867f-8ec05bdf8f15\") " Nov 03 00:33:47.702110 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.701949 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8769d72a-8592-4bed-867f-8ec05bdf8f15-kube-api-access-qfrdj\" (UniqueName: \"kubernetes.io/projected/8769d72a-8592-4bed-867f-8ec05bdf8f15-kube-api-access-qfrdj\") pod \"8769d72a-8592-4bed-867f-8ec05bdf8f15\" (UID: \"8769d72a-8592-4bed-867f-8ec05bdf8f15\") " Nov 03 00:33:47.702319 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.702294 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8769d72a-8592-4bed-867f-8ec05bdf8f15-plugins-config-volume" pod "8769d72a-8592-4bed-867f-8ec05bdf8f15" (UID: "8769d72a-8592-4bed-867f-8ec05bdf8f15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:47.702555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.702533 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8769d72a-8592-4bed-867f-8ec05bdf8f15-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:47.704304 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.704278 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8769d72a-8592-4bed-867f-8ec05bdf8f15-kube-api-access-qfrdj" pod "8769d72a-8592-4bed-867f-8ec05bdf8f15" (UID: "8769d72a-8592-4bed-867f-8ec05bdf8f15"). InnerVolumeSpecName "kube-api-access-qfrdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:47.724923 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.724901 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:33:47.803740 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.803706 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d539e6a6-1183-4598-908d-8a356f55f3d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d539e6a6-1183-4598-908d-8a356f55f3d0-plugins-config-volume\") pod \"d539e6a6-1183-4598-908d-8a356f55f3d0\" (UID: \"d539e6a6-1183-4598-908d-8a356f55f3d0\") " Nov 03 00:33:47.804008 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.803990 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d539e6a6-1183-4598-908d-8a356f55f3d0-kube-api-access-z5lf7\" (UniqueName: \"kubernetes.io/projected/d539e6a6-1183-4598-908d-8a356f55f3d0-kube-api-access-z5lf7\") pod \"d539e6a6-1183-4598-908d-8a356f55f3d0\" (UID: \"d539e6a6-1183-4598-908d-8a356f55f3d0\") " Nov 03 00:33:47.804104 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.804080 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d539e6a6-1183-4598-908d-8a356f55f3d0-plugins-config-volume" pod "d539e6a6-1183-4598-908d-8a356f55f3d0" (UID: "d539e6a6-1183-4598-908d-8a356f55f3d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:47.804433 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.804402 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfrdj\" (UniqueName: \"kubernetes.io/projected/8769d72a-8592-4bed-867f-8ec05bdf8f15-kube-api-access-qfrdj\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:47.804433 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.804431 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d539e6a6-1183-4598-908d-8a356f55f3d0-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:47.806312 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.806285 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d539e6a6-1183-4598-908d-8a356f55f3d0-kube-api-access-z5lf7" pod "d539e6a6-1183-4598-908d-8a356f55f3d0" (UID: "d539e6a6-1183-4598-908d-8a356f55f3d0"). InnerVolumeSpecName "kube-api-access-z5lf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:47.917363 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:47.916560 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5lf7\" (UniqueName: \"kubernetes.io/projected/d539e6a6-1183-4598-908d-8a356f55f3d0-kube-api-access-z5lf7\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:48.078422 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:48.078365 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt"] Nov 03 00:33:48.081466 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:48.081442 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-38f5a3-649f78c946-5dprt"] Nov 03 00:33:48.641619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:48.641545 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" Nov 03 00:33:48.641619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:48.641582 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj" event={"ID":"d539e6a6-1183-4598-908d-8a356f55f3d0","Type":"ContainerDied","Data":"2522eb3fb1946c45a4658e81d2d684603490c052901baf8e3f758dca262496b7"} Nov 03 00:33:48.642133 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:48.641646 4304 scope.go:122] "RemoveContainer" containerID="d229f1459ba4deae15067e149f61d8e5c94c7fa3be7c1756525371dbc7cd0cc9" Nov 03 00:33:48.804950 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:48.804921 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj"] Nov 03 00:33:48.807792 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:48.807753 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-7cb96cc55-mzkhj"] Nov 03 00:33:48.830142 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:48.830108 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8769d72a-8592-4bed-867f-8ec05bdf8f15" path="/var/lib/kubelet/pods/8769d72a-8592-4bed-867f-8ec05bdf8f15/volumes" Nov 03 00:33:48.830495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:48.830474 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d539e6a6-1183-4598-908d-8a356f55f3d0" path="/var/lib/kubelet/pods/d539e6a6-1183-4598-908d-8a356f55f3d0/volumes" Nov 03 00:33:58.657486 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:33:58.657447 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:33:59.635324 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.635283 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p"] Nov 03 00:33:59.635880 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.635863 4304 cpu_manager.go:414] "containerMap: removing container" podUID="d539e6a6-1183-4598-908d-8a356f55f3d0" containerName="payload-pre-processing" Nov 03 00:33:59.635942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.635880 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d539e6a6-1183-4598-908d-8a356f55f3d0" containerName="payload-pre-processing" Nov 03 00:33:59.635942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.635904 4304 cpu_manager.go:414] "containerMap: removing container" podUID="0622afbb-ae90-40fe-b918-910e6a035f85" containerName="payload-processing" Nov 03 00:33:59.635942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.635911 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0622afbb-ae90-40fe-b918-910e6a035f85" containerName="payload-processing" Nov 03 00:33:59.635942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.635926 4304 cpu_manager.go:414] "containerMap: removing container" podUID="8769d72a-8592-4bed-867f-8ec05bdf8f15" containerName="payload-processing" Nov 03 00:33:59.635942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.635931 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8769d72a-8592-4bed-867f-8ec05bdf8f15" containerName="payload-processing" Nov 03 00:33:59.636094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.636007 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d539e6a6-1183-4598-908d-8a356f55f3d0" containerName="payload-pre-processing" Nov 03 00:33:59.636094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.636016 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0622afbb-ae90-40fe-b918-910e6a035f85" containerName="payload-processing" Nov 03 00:33:59.636094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.636024 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8769d72a-8592-4bed-867f-8ec05bdf8f15" containerName="payload-processing" Nov 03 00:33:59.640462 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.640441 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:33:59.673304 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.673267 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgj89\" (UniqueName: \"kubernetes.io/projected/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-kube-api-access-hgj89\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p\" (UID: \"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:33:59.673628 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.673451 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p\" (UID: \"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:33:59.696378 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.696352 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p"] Nov 03 00:33:59.742041 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.741997 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8c9581\"" Nov 03 00:33:59.742212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.742192 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8c9581-dockercfg-r98b2\"" Nov 03 00:33:59.774605 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.774574 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p\" (UID: \"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:33:59.774833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.774649 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgj89\" (UniqueName: \"kubernetes.io/projected/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-kube-api-access-hgj89\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p\" (UID: \"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:33:59.775275 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.775256 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p\" (UID: \"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:33:59.837769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.837738 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgj89\" (UniqueName: \"kubernetes.io/projected/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-kube-api-access-hgj89\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p\" (UID: \"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:33:59.955074 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:33:59.955034 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:34:00.331040 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:00.331012 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p"] Nov 03 00:34:00.332594 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:34:00.332569 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25356aa_c1d3_4112_b54b_5c0f6c89e8c8.slice/crio-8cb5fb5d2ce86f14e2895928336a60c6e056e7aa905c95e8ebbb1cd49ca9d3ef WatchSource:0}: Error finding container 8cb5fb5d2ce86f14e2895928336a60c6e056e7aa905c95e8ebbb1cd49ca9d3ef: Status 404 returned error can't find the container with id 8cb5fb5d2ce86f14e2895928336a60c6e056e7aa905c95e8ebbb1cd49ca9d3ef Nov 03 00:34:00.732106 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:00.732071 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" event={"ID":"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8","Type":"ContainerStarted","Data":"8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd"} Nov 03 00:34:00.732517 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:00.732120 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" event={"ID":"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8","Type":"ContainerStarted","Data":"8cb5fb5d2ce86f14e2895928336a60c6e056e7aa905c95e8ebbb1cd49ca9d3ef"} Nov 03 00:34:00.732517 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:00.732216 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:34:00.849994 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:00.849939 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" podStartSLOduration=1.849921019 podStartE2EDuration="1.849921019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:00.847462388 +0000 UTC m=+1724.131020063" watchObservedRunningTime="2026-11-03 00:34:00.849921019 +0000 UTC m=+1724.133478695" Nov 03 00:34:11.740542 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:11.740507 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:34:16.677008 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.676969 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x"] Nov 03 00:34:16.680936 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.680919 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:34:16.734087 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.734041 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3df441\"" Nov 03 00:34:16.767057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.767016 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x\" (UID: \"d9cb7dee-1a14-48c2-bba3-19af10d4c28b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:34:16.767256 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.767128 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mrtp\" (UniqueName: \"kubernetes.io/projected/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-kube-api-access-2mrtp\") pod \"payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x\" (UID: \"d9cb7dee-1a14-48c2-bba3-19af10d4c28b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:34:16.783244 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.783221 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3df441-dockercfg-xmncr\"" Nov 03 00:34:16.786445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.786423 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x"] Nov 03 00:34:16.868003 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.867973 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x\" (UID: \"d9cb7dee-1a14-48c2-bba3-19af10d4c28b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:34:16.868175 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.868021 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mrtp\" (UniqueName: \"kubernetes.io/projected/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-kube-api-access-2mrtp\") pod \"payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x\" (UID: \"d9cb7dee-1a14-48c2-bba3-19af10d4c28b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:34:16.920014 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.919986 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3df441\"" Nov 03 00:34:16.928801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.928724 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x\" (UID: \"d9cb7dee-1a14-48c2-bba3-19af10d4c28b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:34:16.932274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:16.932256 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mrtp\" (UniqueName: \"kubernetes.io/projected/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-kube-api-access-2mrtp\") pod \"payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x\" (UID: \"d9cb7dee-1a14-48c2-bba3-19af10d4c28b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:34:17.045647 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.045613 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3df441-dockercfg-xmncr\"" Nov 03 00:34:17.054996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.054965 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:34:17.435508 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.435479 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x"] Nov 03 00:34:17.436697 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:34:17.436673 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cb7dee_1a14_48c2_bba3_19af10d4c28b.slice/crio-9f088ff7dc0148d710be7e0e5438f1d76de44ed30920f112400cde0db51d04e5 WatchSource:0}: Error finding container 9f088ff7dc0148d710be7e0e5438f1d76de44ed30920f112400cde0db51d04e5: Status 404 returned error can't find the container with id 9f088ff7dc0148d710be7e0e5438f1d76de44ed30920f112400cde0db51d04e5 Nov 03 00:34:17.480097 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.480073 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4"] Nov 03 00:34:17.484424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.484398 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:34:17.549391 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.549317 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4"] Nov 03 00:34:17.574283 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.574248 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42gx\" (UniqueName: \"kubernetes.io/projected/df822e75-e797-4eaf-af86-88f3af76deed-kube-api-access-m42gx\") pod \"payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4\" (UID: \"df822e75-e797-4eaf-af86-88f3af76deed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:34:17.574481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.574384 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df822e75-e797-4eaf-af86-88f3af76deed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4\" (UID: \"df822e75-e797-4eaf-af86-88f3af76deed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:34:17.675624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.675591 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42gx\" (UniqueName: \"kubernetes.io/projected/df822e75-e797-4eaf-af86-88f3af76deed-kube-api-access-m42gx\") pod \"payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4\" (UID: \"df822e75-e797-4eaf-af86-88f3af76deed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:34:17.675818 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.675639 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df822e75-e797-4eaf-af86-88f3af76deed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4\" (UID: \"df822e75-e797-4eaf-af86-88f3af76deed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:34:17.676415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.676396 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df822e75-e797-4eaf-af86-88f3af76deed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4\" (UID: \"df822e75-e797-4eaf-af86-88f3af76deed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:34:17.731833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.731804 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42gx\" (UniqueName: \"kubernetes.io/projected/df822e75-e797-4eaf-af86-88f3af76deed-kube-api-access-m42gx\") pod \"payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4\" (UID: \"df822e75-e797-4eaf-af86-88f3af76deed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:34:17.804304 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.804232 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:34:17.848366 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.848322 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" event={"ID":"d9cb7dee-1a14-48c2-bba3-19af10d4c28b","Type":"ContainerStarted","Data":"45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789"} Nov 03 00:34:17.848546 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.848381 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" event={"ID":"d9cb7dee-1a14-48c2-bba3-19af10d4c28b","Type":"ContainerStarted","Data":"9f088ff7dc0148d710be7e0e5438f1d76de44ed30920f112400cde0db51d04e5"} Nov 03 00:34:17.848619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.848555 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:34:17.963449 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:17.963400 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" podStartSLOduration=1.9633831819999998 podStartE2EDuration="1.963383182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:17.961874713 +0000 UTC m=+1741.245432387" watchObservedRunningTime="2026-11-03 00:34:17.963383182 +0000 UTC m=+1741.246940857" Nov 03 00:34:18.181644 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:18.181621 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4"] Nov 03 00:34:18.182890 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:34:18.182864 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf822e75_e797_4eaf_af86_88f3af76deed.slice/crio-e880842a7776541683d68fbf1efc9815bb62e3d3d9ada2a91bfd30b0704d0884 WatchSource:0}: Error finding container e880842a7776541683d68fbf1efc9815bb62e3d3d9ada2a91bfd30b0704d0884: Status 404 returned error can't find the container with id e880842a7776541683d68fbf1efc9815bb62e3d3d9ada2a91bfd30b0704d0884 Nov 03 00:34:18.856266 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:18.856236 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" event={"ID":"df822e75-e797-4eaf-af86-88f3af76deed","Type":"ContainerStarted","Data":"b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701"} Nov 03 00:34:18.856685 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:18.856292 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" event={"ID":"df822e75-e797-4eaf-af86-88f3af76deed","Type":"ContainerStarted","Data":"e880842a7776541683d68fbf1efc9815bb62e3d3d9ada2a91bfd30b0704d0884"} Nov 03 00:34:18.856685 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:18.856331 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:34:18.971353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:18.971305 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" podStartSLOduration=1.971290854 podStartE2EDuration="1.971290854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:18.970570604 +0000 UTC m=+1742.254128278" watchObservedRunningTime="2026-11-03 00:34:18.971290854 +0000 UTC m=+1742.254848528" Nov 03 00:34:28.654945 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:34:28.654905 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:34:28.858452 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:28.858423 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:34:29.864714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:29.864637 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:34:40.587196 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:40.587164 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m"] Nov 03 00:34:40.587606 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:40.587431 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" podUID="d68711b8-ca31-416b-950f-26b3d43f73ad" containerName="istio-proxy" containerID="cri-o://0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4" gracePeriod=30 Nov 03 00:34:40.592125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:40.592103 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd"] Nov 03 00:34:40.592337 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:40.592316 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" podUID="8137f3ca-7b7e-40e4-95e0-c97518557954" containerName="istio-proxy" containerID="cri-o://366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7" gracePeriod=30 Nov 03 00:34:45.890322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.890291 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:34:45.927992 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.927961 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:34:45.964307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964275 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-socket\") pod \"d68711b8-ca31-416b-950f-26b3d43f73ad\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " Nov 03 00:34:45.964307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964309 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-data\") pod \"d68711b8-ca31-416b-950f-26b3d43f73ad\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " Nov 03 00:34:45.964612 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964332 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-envoy\") pod \"d68711b8-ca31-416b-950f-26b3d43f73ad\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " Nov 03 00:34:45.964612 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964389 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-credential-socket\") pod \"d68711b8-ca31-416b-950f-26b3d43f73ad\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " Nov 03 00:34:45.964612 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964424 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-token\" (UniqueName: \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-token\") pod \"d68711b8-ca31-416b-950f-26b3d43f73ad\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " Nov 03 00:34:45.964612 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964552 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-podinfo\") pod \"d68711b8-ca31-416b-950f-26b3d43f73ad\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " Nov 03 00:34:45.964612 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964591 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-socket" pod "d68711b8-ca31-416b-950f-26b3d43f73ad" (UID: "d68711b8-ca31-416b-950f-26b3d43f73ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.964924 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964601 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-kube-api-access-gf2wr\" (UniqueName: \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-kube-api-access-gf2wr\") pod \"d68711b8-ca31-416b-950f-26b3d43f73ad\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " Nov 03 00:34:45.964924 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964616 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-data" pod "d68711b8-ca31-416b-950f-26b3d43f73ad" (UID: "d68711b8-ca31-416b-950f-26b3d43f73ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.964924 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964724 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-credential-socket" pod "d68711b8-ca31-416b-950f-26b3d43f73ad" (UID: "d68711b8-ca31-416b-950f-26b3d43f73ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.964924 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964759 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68711b8-ca31-416b-950f-26b3d43f73ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68711b8-ca31-416b-950f-26b3d43f73ad-istiod-ca-cert\") pod \"d68711b8-ca31-416b-950f-26b3d43f73ad\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " Nov 03 00:34:45.964924 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.964851 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-certs\") pod \"d68711b8-ca31-416b-950f-26b3d43f73ad\" (UID: \"d68711b8-ca31-416b-950f-26b3d43f73ad\") " Nov 03 00:34:45.965235 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.965210 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-certs" pod "d68711b8-ca31-416b-950f-26b3d43f73ad" (UID: "d68711b8-ca31-416b-950f-26b3d43f73ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.965481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.965242 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68711b8-ca31-416b-950f-26b3d43f73ad-istiod-ca-cert" pod "d68711b8-ca31-416b-950f-26b3d43f73ad" (UID: "d68711b8-ca31-416b-950f-26b3d43f73ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:45.965574 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.965553 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68711b8-ca31-416b-950f-26b3d43f73ad-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.965656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.965579 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.965656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.965594 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.965656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.965609 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.965656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.965622 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.967016 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.966992 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-podinfo" pod "d68711b8-ca31-416b-950f-26b3d43f73ad" (UID: "d68711b8-ca31-416b-950f-26b3d43f73ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:34:45.967016 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.966999 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-envoy" pod "d68711b8-ca31-416b-950f-26b3d43f73ad" (UID: "d68711b8-ca31-416b-950f-26b3d43f73ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.967155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.967042 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-token" pod "d68711b8-ca31-416b-950f-26b3d43f73ad" (UID: "d68711b8-ca31-416b-950f-26b3d43f73ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:45.967155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:45.967069 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-kube-api-access-gf2wr" pod "d68711b8-ca31-416b-950f-26b3d43f73ad" (UID: "d68711b8-ca31-416b-950f-26b3d43f73ad"). InnerVolumeSpecName "kube-api-access-gf2wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:46.039885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.039840 4304 generic.go:356] "Generic (PLEG): container finished" podID="d68711b8-ca31-416b-950f-26b3d43f73ad" containerID="0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4" exitCode=0 Nov 03 00:34:46.040069 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.039900 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" Nov 03 00:34:46.040069 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.039923 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" event={"ID":"d68711b8-ca31-416b-950f-26b3d43f73ad","Type":"ContainerDied","Data":"0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4"} Nov 03 00:34:46.040192 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.040073 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m" event={"ID":"d68711b8-ca31-416b-950f-26b3d43f73ad","Type":"ContainerDied","Data":"99555f2310cbcc9c2f3f8ce83e2724c75a65376ecee300f7540dfce0bf994201"} Nov 03 00:34:46.040192 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.040111 4304 scope.go:122] "RemoveContainer" containerID="0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4" Nov 03 00:34:46.042155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.041977 4304 generic.go:356] "Generic (PLEG): container finished" podID="8137f3ca-7b7e-40e4-95e0-c97518557954" containerID="366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7" exitCode=0 Nov 03 00:34:46.042155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.042026 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" Nov 03 00:34:46.042155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.042073 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" event={"ID":"8137f3ca-7b7e-40e4-95e0-c97518557954","Type":"ContainerDied","Data":"366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7"} Nov 03 00:34:46.042155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.042113 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd" event={"ID":"8137f3ca-7b7e-40e4-95e0-c97518557954","Type":"ContainerDied","Data":"5e85a44f1b8ce75c10a7c68caae60f85336167190b46e86637c81a0fe15f28a7"} Nov 03 00:34:46.054257 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.054240 4304 scope.go:122] "RemoveContainer" containerID="0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4" Nov 03 00:34:46.054550 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:34:46.054522 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4\": container with ID starting with 0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4 not found: ID does not exist" containerID="0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4" Nov 03 00:34:46.054649 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.054560 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4"} err="failed to get container status \"0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4\": rpc error: code = NotFound desc = could not find container \"0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4\": container with ID starting with 0490d4bd24d0bdd7a706a763c7a0b35b719908990b80798a22410f924bd393d4 not found: ID does not exist" Nov 03 00:34:46.054649 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.054585 4304 scope.go:122] "RemoveContainer" containerID="366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7" Nov 03 00:34:46.066175 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.066148 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-certs\") pod \"8137f3ca-7b7e-40e4-95e0-c97518557954\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " Nov 03 00:34:46.066175 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.066177 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-data\") pod \"8137f3ca-7b7e-40e4-95e0-c97518557954\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " Nov 03 00:34:46.066373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.066252 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-token\" (UniqueName: \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-token\") pod \"8137f3ca-7b7e-40e4-95e0-c97518557954\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " Nov 03 00:34:46.066373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.066338 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8137f3ca-7b7e-40e4-95e0-c97518557954-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8137f3ca-7b7e-40e4-95e0-c97518557954-istiod-ca-cert\") pod \"8137f3ca-7b7e-40e4-95e0-c97518557954\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " Nov 03 00:34:46.066606 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.066431 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-credential-socket\") pod \"8137f3ca-7b7e-40e4-95e0-c97518557954\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " Nov 03 00:34:46.066606 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.066452 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-certs" pod "8137f3ca-7b7e-40e4-95e0-c97518557954" (UID: "8137f3ca-7b7e-40e4-95e0-c97518557954"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:46.066606 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.066481 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-data" pod "8137f3ca-7b7e-40e4-95e0-c97518557954" (UID: "8137f3ca-7b7e-40e4-95e0-c97518557954"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:46.066606 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.066530 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-kube-api-access-4m48w\" (UniqueName: \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-kube-api-access-4m48w\") pod \"8137f3ca-7b7e-40e4-95e0-c97518557954\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " Nov 03 00:34:46.066606 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.066562 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-podinfo\") pod \"8137f3ca-7b7e-40e4-95e0-c97518557954\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " Nov 03 00:34:46.066887 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.066658 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-socket\") pod \"8137f3ca-7b7e-40e4-95e0-c97518557954\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " Nov 03 00:34:46.066887 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.066714 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-envoy\") pod \"8137f3ca-7b7e-40e4-95e0-c97518557954\" (UID: \"8137f3ca-7b7e-40e4-95e0-c97518557954\") " Nov 03 00:34:46.067306 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.067282 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.067394 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.067310 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.067394 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.067325 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.067394 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.067337 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.067394 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.067350 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68711b8-ca31-416b-950f-26b3d43f73ad-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.067394 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.067363 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf2wr\" (UniqueName: \"kubernetes.io/projected/d68711b8-ca31-416b-950f-26b3d43f73ad-kube-api-access-gf2wr\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.067674 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.067656 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-credential-socket" pod "8137f3ca-7b7e-40e4-95e0-c97518557954" (UID: "8137f3ca-7b7e-40e4-95e0-c97518557954"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:46.067745 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.067726 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8137f3ca-7b7e-40e4-95e0-c97518557954-istiod-ca-cert" pod "8137f3ca-7b7e-40e4-95e0-c97518557954" (UID: "8137f3ca-7b7e-40e4-95e0-c97518557954"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:46.067983 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.067960 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-socket" pod "8137f3ca-7b7e-40e4-95e0-c97518557954" (UID: "8137f3ca-7b7e-40e4-95e0-c97518557954"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:46.069113 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.068560 4304 scope.go:122] "RemoveContainer" containerID="366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7" Nov 03 00:34:46.069113 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:34:46.068998 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7\": container with ID starting with 366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7 not found: ID does not exist" containerID="366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7" Nov 03 00:34:46.069113 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.069026 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7"} err="failed to get container status \"366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7\": rpc error: code = NotFound desc = could not find container \"366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7\": container with ID starting with 366870fda698c4774ff772fa16e7c31135fcf01e1a26aa42f9ae902b3e9af4f7 not found: ID does not exist" Nov 03 00:34:46.070356 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.070331 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-token" pod "8137f3ca-7b7e-40e4-95e0-c97518557954" (UID: "8137f3ca-7b7e-40e4-95e0-c97518557954"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:46.070496 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.070421 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-kube-api-access-4m48w" pod "8137f3ca-7b7e-40e4-95e0-c97518557954" (UID: "8137f3ca-7b7e-40e4-95e0-c97518557954"). InnerVolumeSpecName "kube-api-access-4m48w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:46.070644 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.070611 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-envoy" pod "8137f3ca-7b7e-40e4-95e0-c97518557954" (UID: "8137f3ca-7b7e-40e4-95e0-c97518557954"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:46.070856 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.070834 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-podinfo" pod "8137f3ca-7b7e-40e4-95e0-c97518557954" (UID: "8137f3ca-7b7e-40e4-95e0-c97518557954"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:34:46.169292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.169257 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8137f3ca-7b7e-40e4-95e0-c97518557954-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.169292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.169296 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.169482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.169311 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m48w\" (UniqueName: \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-kube-api-access-4m48w\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.169482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.169328 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.169482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.169342 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.169482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.169356 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.169482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.169368 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8137f3ca-7b7e-40e4-95e0-c97518557954-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.171002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.170977 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m"] Nov 03 00:34:46.178045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.178018 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-v5f2m"] Nov 03 00:34:46.466858 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.466746 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd"] Nov 03 00:34:46.472710 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.472682 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5crvd"] Nov 03 00:34:46.829959 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.829886 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8137f3ca-7b7e-40e4-95e0-c97518557954" path="/var/lib/kubelet/pods/8137f3ca-7b7e-40e4-95e0-c97518557954/volumes" Nov 03 00:34:46.830382 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:46.830366 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68711b8-ca31-416b-950f-26b3d43f73ad" path="/var/lib/kubelet/pods/d68711b8-ca31-416b-950f-26b3d43f73ad/volumes" Nov 03 00:34:55.459365 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.459327 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282"] Nov 03 00:34:55.460000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.459585 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" podUID="1c6fec01-18e7-4980-8cc2-7cf2344b0f6f" containerName="maas-api" containerID="cri-o://8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82" gracePeriod=30 Nov 03 00:34:55.742237 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.742209 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:34:55.785597 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.785568 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-openshift-service-ca\") pod \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " Nov 03 00:34:55.785806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.785635 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-metrics-certs\") pod \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " Nov 03 00:34:55.785806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.785664 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-kube-api-access-mpqrf\" (UniqueName: \"kubernetes.io/projected/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-kube-api-access-mpqrf\") pod \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " Nov 03 00:34:55.785806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.785718 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-maas-api-tls\") pod \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\" (UID: \"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f\") " Nov 03 00:34:55.786081 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.786050 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-openshift-service-ca" pod "1c6fec01-18e7-4980-8cc2-7cf2344b0f6f" (UID: "1c6fec01-18e7-4980-8cc2-7cf2344b0f6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:55.786265 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.786249 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:55.787968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.787940 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-maas-api-tls" pod "1c6fec01-18e7-4980-8cc2-7cf2344b0f6f" (UID: "1c6fec01-18e7-4980-8cc2-7cf2344b0f6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:34:55.788065 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.787971 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-kube-api-access-mpqrf" pod "1c6fec01-18e7-4980-8cc2-7cf2344b0f6f" (UID: "1c6fec01-18e7-4980-8cc2-7cf2344b0f6f"). InnerVolumeSpecName "kube-api-access-mpqrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:55.788065 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.788023 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-metrics-certs" pod "1c6fec01-18e7-4980-8cc2-7cf2344b0f6f" (UID: "1c6fec01-18e7-4980-8cc2-7cf2344b0f6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:34:55.887939 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.887904 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpqrf\" (UniqueName: \"kubernetes.io/projected/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-kube-api-access-mpqrf\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:55.887939 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.887937 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:55.888158 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:55.887949 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:56.084149 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.084071 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p"] Nov 03 00:34:56.084321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.084272 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" podUID="d25356aa-c1d3-4112-b54b-5c0f6c89e8c8" containerName="payload-pre-processing" containerID="cri-o://8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd" gracePeriod=30 Nov 03 00:34:56.114498 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.114470 4304 generic.go:356] "Generic (PLEG): container finished" podID="1c6fec01-18e7-4980-8cc2-7cf2344b0f6f" containerID="8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82" exitCode=0 Nov 03 00:34:56.114631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.114516 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" Nov 03 00:34:56.114631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.114551 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" event={"ID":"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f","Type":"ContainerDied","Data":"8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82"} Nov 03 00:34:56.114631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.114601 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282" event={"ID":"1c6fec01-18e7-4980-8cc2-7cf2344b0f6f","Type":"ContainerDied","Data":"9df14bf68f062811b316e43bb0ee38fa9de0f19ce125b71b116dcf55e0bbe358"} Nov 03 00:34:56.114631 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.114624 4304 scope.go:122] "RemoveContainer" containerID="8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82" Nov 03 00:34:56.209211 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.209185 4304 scope.go:122] "RemoveContainer" containerID="8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82" Nov 03 00:34:56.209594 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:34:56.209557 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82\": container with ID starting with 8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82 not found: ID does not exist" containerID="8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82" Nov 03 00:34:56.209699 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.209600 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82"} err="failed to get container status \"8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82\": rpc error: code = NotFound desc = could not find container \"8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82\": container with ID starting with 8b0d370185cab117e38b3fc5ec1a2fb79651306bbb94927fbc55898a445a7d82 not found: ID does not exist" Nov 03 00:34:56.310431 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.310397 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282"] Nov 03 00:34:56.316647 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.316614 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8c9581-c7c95db45-mf282"] Nov 03 00:34:56.330831 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.330808 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:34:56.394557 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.394536 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-plugins-config-volume\") pod \"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8\" (UID: \"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8\") " Nov 03 00:34:56.394727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.394606 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-kube-api-access-hgj89\" (UniqueName: \"kubernetes.io/projected/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-kube-api-access-hgj89\") pod \"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8\" (UID: \"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8\") " Nov 03 00:34:56.394955 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.394930 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-plugins-config-volume" pod "d25356aa-c1d3-4112-b54b-5c0f6c89e8c8" (UID: "d25356aa-c1d3-4112-b54b-5c0f6c89e8c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:56.396831 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.396809 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-kube-api-access-hgj89" pod "d25356aa-c1d3-4112-b54b-5c0f6c89e8c8" (UID: "d25356aa-c1d3-4112-b54b-5c0f6c89e8c8"). InnerVolumeSpecName "kube-api-access-hgj89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:56.495708 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.495679 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:56.496056 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.495711 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgj89\" (UniqueName: \"kubernetes.io/projected/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8-kube-api-access-hgj89\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:56.830104 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:56.830026 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6fec01-18e7-4980-8cc2-7cf2344b0f6f" path="/var/lib/kubelet/pods/1c6fec01-18e7-4980-8cc2-7cf2344b0f6f/volumes" Nov 03 00:34:57.123098 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:57.123009 4304 generic.go:356] "Generic (PLEG): container finished" podID="d25356aa-c1d3-4112-b54b-5c0f6c89e8c8" containerID="8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd" exitCode=0 Nov 03 00:34:57.123098 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:57.123063 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" Nov 03 00:34:57.123322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:57.123095 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" event={"ID":"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8","Type":"ContainerDied","Data":"8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd"} Nov 03 00:34:57.123322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:57.123147 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p" event={"ID":"d25356aa-c1d3-4112-b54b-5c0f6c89e8c8","Type":"ContainerDied","Data":"8cb5fb5d2ce86f14e2895928336a60c6e056e7aa905c95e8ebbb1cd49ca9d3ef"} Nov 03 00:34:57.123322 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:57.123181 4304 scope.go:122] "RemoveContainer" containerID="8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd" Nov 03 00:34:57.132563 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:57.132539 4304 scope.go:122] "RemoveContainer" containerID="8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd" Nov 03 00:34:57.132876 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:34:57.132851 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd\": container with ID starting with 8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd not found: ID does not exist" containerID="8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd" Nov 03 00:34:57.132930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:57.132887 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd"} err="failed to get container status \"8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd\": rpc error: code = NotFound desc = could not find container \"8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd\": container with ID starting with 8116e80eafc679d86f2c4b3e588f8099ba7fa5c6c607a949aa726ae93e31adfd not found: ID does not exist" Nov 03 00:34:57.286629 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:57.286597 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p"] Nov 03 00:34:57.292473 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:57.292449 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-7b494d77c5-d847p"] Nov 03 00:34:58.480618 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:58.480579 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw"] Nov 03 00:34:58.481072 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:58.480813 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerName="istio-proxy" containerID="cri-o://c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f" gracePeriod=30 Nov 03 00:34:58.655857 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:34:58.655822 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:34:58.833285 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:34:58.833199 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25356aa-c1d3-4112-b54b-5c0f6c89e8c8" path="/var/lib/kubelet/pods/d25356aa-c1d3-4112-b54b-5c0f6c89e8c8/volumes" Nov 03 00:35:02.923622 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:02.923594 4304 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:35:02.924045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:02.923640 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:35:03.789793 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.788381 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:35:03.881438 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881403 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6794e0ff-4c6d-48d0-9982-6241c13b216d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6794e0ff-4c6d-48d0-9982-6241c13b216d-istiod-ca-cert\") pod \"6794e0ff-4c6d-48d0-9982-6241c13b216d\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " Nov 03 00:35:03.881645 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881445 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-kube-api-access-28jfw\" (UniqueName: \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-kube-api-access-28jfw\") pod \"6794e0ff-4c6d-48d0-9982-6241c13b216d\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " Nov 03 00:35:03.881645 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881552 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-certs\") pod \"6794e0ff-4c6d-48d0-9982-6241c13b216d\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " Nov 03 00:35:03.881645 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881600 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-data\") pod \"6794e0ff-4c6d-48d0-9982-6241c13b216d\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " Nov 03 00:35:03.881645 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881635 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-envoy\") pod \"6794e0ff-4c6d-48d0-9982-6241c13b216d\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " Nov 03 00:35:03.881901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881661 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-socket\") pod \"6794e0ff-4c6d-48d0-9982-6241c13b216d\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " Nov 03 00:35:03.881901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881695 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-token\" (UniqueName: \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-token\") pod \"6794e0ff-4c6d-48d0-9982-6241c13b216d\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " Nov 03 00:35:03.881901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881761 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-credential-socket\") pod \"6794e0ff-4c6d-48d0-9982-6241c13b216d\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " Nov 03 00:35:03.881901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881844 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6794e0ff-4c6d-48d0-9982-6241c13b216d-istiod-ca-cert" pod "6794e0ff-4c6d-48d0-9982-6241c13b216d" (UID: "6794e0ff-4c6d-48d0-9982-6241c13b216d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:03.881901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881897 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-podinfo\") pod \"6794e0ff-4c6d-48d0-9982-6241c13b216d\" (UID: \"6794e0ff-4c6d-48d0-9982-6241c13b216d\") " Nov 03 00:35:03.882131 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881918 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-certs" pod "6794e0ff-4c6d-48d0-9982-6241c13b216d" (UID: "6794e0ff-4c6d-48d0-9982-6241c13b216d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:03.882131 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.881928 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-socket" pod "6794e0ff-4c6d-48d0-9982-6241c13b216d" (UID: "6794e0ff-4c6d-48d0-9982-6241c13b216d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:03.882131 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.882106 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-credential-socket" pod "6794e0ff-4c6d-48d0-9982-6241c13b216d" (UID: "6794e0ff-4c6d-48d0-9982-6241c13b216d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:03.882443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.882237 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-data" pod "6794e0ff-4c6d-48d0-9982-6241c13b216d" (UID: "6794e0ff-4c6d-48d0-9982-6241c13b216d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:03.883693 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.882901 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6794e0ff-4c6d-48d0-9982-6241c13b216d-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.883845 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.883709 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.883845 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.883729 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.883845 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.883744 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.883845 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.883759 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.884281 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.884257 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-kube-api-access-28jfw" pod "6794e0ff-4c6d-48d0-9982-6241c13b216d" (UID: "6794e0ff-4c6d-48d0-9982-6241c13b216d"). InnerVolumeSpecName "kube-api-access-28jfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:03.884549 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.884529 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-envoy" pod "6794e0ff-4c6d-48d0-9982-6241c13b216d" (UID: "6794e0ff-4c6d-48d0-9982-6241c13b216d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:03.884549 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.884541 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-podinfo" pod "6794e0ff-4c6d-48d0-9982-6241c13b216d" (UID: "6794e0ff-4c6d-48d0-9982-6241c13b216d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:35:03.884646 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.884559 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-token" pod "6794e0ff-4c6d-48d0-9982-6241c13b216d" (UID: "6794e0ff-4c6d-48d0-9982-6241c13b216d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:03.985116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.985084 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.985116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.985113 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28jfw\" (UniqueName: \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-kube-api-access-28jfw\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.985116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.985125 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.985560 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:03.985133 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6794e0ff-4c6d-48d0-9982-6241c13b216d-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:04.171545 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:04.171504 4304 generic.go:356] "Generic (PLEG): container finished" podID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerID="c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f" exitCode=0 Nov 03 00:35:04.171748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:04.171550 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" Nov 03 00:35:04.171748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:04.171588 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" event={"ID":"6794e0ff-4c6d-48d0-9982-6241c13b216d","Type":"ContainerDied","Data":"c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f"} Nov 03 00:35:04.171748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:04.171652 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw" event={"ID":"6794e0ff-4c6d-48d0-9982-6241c13b216d","Type":"ContainerDied","Data":"4deb2461391afada4d9c2677a9da2a240b279c5cb149a41acce652284b3aaafb"} Nov 03 00:35:04.171748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:04.171683 4304 scope.go:122] "RemoveContainer" containerID="c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f" Nov 03 00:35:04.182518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:04.182503 4304 scope.go:122] "RemoveContainer" containerID="c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f" Nov 03 00:35:04.182769 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:35:04.182752 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f\": container with ID starting with c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f not found: ID does not exist" containerID="c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f" Nov 03 00:35:04.182889 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:04.182793 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f"} err="failed to get container status \"c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f\": rpc error: code = NotFound desc = could not find container \"c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f\": container with ID starting with c0b6a83eb62ca8f37a0e34c031218c145e68fcbabc59a93808366cf4a436028f not found: ID does not exist" Nov 03 00:35:04.337757 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:04.337722 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw"] Nov 03 00:35:04.340671 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:04.340645 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-j7lgw"] Nov 03 00:35:04.828769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:04.828729 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" path="/var/lib/kubelet/pods/6794e0ff-4c6d-48d0-9982-6241c13b216d/volumes" Nov 03 00:35:12.066071 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:12.066043 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6"] Nov 03 00:35:12.178195 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:12.178160 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc"] Nov 03 00:35:13.627327 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.627294 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc"] Nov 03 00:35:13.627871 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.627857 4304 cpu_manager.go:414] "containerMap: removing container" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerName="istio-proxy" Nov 03 00:35:13.627935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.627873 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerName="istio-proxy" Nov 03 00:35:13.627935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.627894 4304 cpu_manager.go:414] "containerMap: removing container" podUID="1c6fec01-18e7-4980-8cc2-7cf2344b0f6f" containerName="maas-api" Nov 03 00:35:13.627935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.627902 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6fec01-18e7-4980-8cc2-7cf2344b0f6f" containerName="maas-api" Nov 03 00:35:13.627935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.627925 4304 cpu_manager.go:414] "containerMap: removing container" podUID="d25356aa-c1d3-4112-b54b-5c0f6c89e8c8" containerName="payload-pre-processing" Nov 03 00:35:13.627935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.627930 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25356aa-c1d3-4112-b54b-5c0f6c89e8c8" containerName="payload-pre-processing" Nov 03 00:35:13.628092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.627943 4304 cpu_manager.go:414] "containerMap: removing container" podUID="d68711b8-ca31-416b-950f-26b3d43f73ad" containerName="istio-proxy" Nov 03 00:35:13.628092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.627950 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68711b8-ca31-416b-950f-26b3d43f73ad" containerName="istio-proxy" Nov 03 00:35:13.628092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.627958 4304 cpu_manager.go:414] "containerMap: removing container" podUID="8137f3ca-7b7e-40e4-95e0-c97518557954" containerName="istio-proxy" Nov 03 00:35:13.628092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.627963 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8137f3ca-7b7e-40e4-95e0-c97518557954" containerName="istio-proxy" Nov 03 00:35:13.628092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.628050 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6fec01-18e7-4980-8cc2-7cf2344b0f6f" containerName="maas-api" Nov 03 00:35:13.628092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.628062 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25356aa-c1d3-4112-b54b-5c0f6c89e8c8" containerName="payload-pre-processing" Nov 03 00:35:13.628092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.628075 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68711b8-ca31-416b-950f-26b3d43f73ad" containerName="istio-proxy" Nov 03 00:35:13.628092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.628083 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8137f3ca-7b7e-40e4-95e0-c97518557954" containerName="istio-proxy" Nov 03 00:35:13.628092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.628091 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6794e0ff-4c6d-48d0-9982-6241c13b216d" containerName="istio-proxy" Nov 03 00:35:13.631397 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.631379 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.676882 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.676852 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq"] Nov 03 00:35:13.680747 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.680727 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc"] Nov 03 00:35:13.680917 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.680867 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.684302 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.684282 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.722999 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.722974 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zx4cj\"" Nov 03 00:35:13.733292 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.733269 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc"] Nov 03 00:35:13.815855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.815813 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.816050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.815892 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90a54d0-dd85-433d-a372-fecd7d8c5106-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.816050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.815924 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.816050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.815959 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd16683-a7b5-4f26-8a6f-2a3160558426-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.816050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.815990 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.816050 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816023 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.816332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816054 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.816332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816081 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.816332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816119 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.816332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816162 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.816332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816196 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.816332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816242 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.816332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816274 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a413a39-1efa-4c79-8a34-d40431400818-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.816688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816359 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.816688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816423 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.816688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816478 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.816688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816577 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.816688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816614 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.816688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816663 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.817058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816697 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86p9x\" (UniqueName: \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-kube-api-access-86p9x\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.817058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816733 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.817058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816791 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2pf2\" (UniqueName: \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-kube-api-access-w2pf2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.817058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816827 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.817058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.816908 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a413a39-1efa-4c79-8a34-d40431400818-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.817321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.817142 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqp26\" (UniqueName: \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-kube-api-access-vqp26\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.817412 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.817394 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.817556 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.817539 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.819611 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.819585 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq"] Nov 03 00:35:13.834816 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.834769 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc"] Nov 03 00:35:13.918924 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.918895 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.918924 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.918927 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.919184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.918953 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.919184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.918971 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86p9x\" (UniqueName: \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-kube-api-access-86p9x\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.919184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919000 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.919184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919025 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2pf2\" (UniqueName: \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-kube-api-access-w2pf2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.919184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919048 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.919480 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919378 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.919480 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919450 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.919586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919515 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a413a39-1efa-4c79-8a34-d40431400818-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.919586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919565 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqp26\" (UniqueName: \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-kube-api-access-vqp26\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.919699 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919602 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.919699 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919640 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.919699 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919674 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.919908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919725 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90a54d0-dd85-433d-a372-fecd7d8c5106-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.919908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919755 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.919908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919798 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd16683-a7b5-4f26-8a6f-2a3160558426-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.919908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919825 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.919908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919853 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.919908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919877 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.919908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919902 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.920318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919933 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.920318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919966 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.920318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919990 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.920318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.919993 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.920318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.920241 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.920318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.920250 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a413a39-1efa-4c79-8a34-d40431400818-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.920626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.920445 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.920626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.920536 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.920847 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.920805 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.920939 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.920857 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.920939 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.920865 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.920939 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.920910 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a413a39-1efa-4c79-8a34-d40431400818-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.921107 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.921010 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90a54d0-dd85-433d-a372-fecd7d8c5106-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.921164 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.921111 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.921217 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.921181 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.921270 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.921237 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.921329 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.921288 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.921548 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.921524 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.921893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.921644 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.921893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.921885 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.922081 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.922059 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.922154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.922099 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd16683-a7b5-4f26-8a6f-2a3160558426-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.922213 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.922151 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.923274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.923257 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.924035 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.924014 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a413a39-1efa-4c79-8a34-d40431400818-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.924150 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.924053 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.924240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.924219 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.976168 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.976135 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.976525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.976493 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.976948 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.976926 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86p9x\" (UniqueName: \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-kube-api-access-86p9x\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:13.977041 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.976953 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqp26\" (UniqueName: \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-kube-api-access-vqp26\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:13.977041 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.976928 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.977041 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.976960 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2pf2\" (UniqueName: \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-kube-api-access-w2pf2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:13.996595 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:13.996575 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:14.002851 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.002830 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:14.245019 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.244938 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:14.388629 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.388599 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq"] Nov 03 00:35:14.389469 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:35:14.389445 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90a54d0_dd85_433d_a372_fecd7d8c5106.slice/crio-138ee7c2a506cdecb11677a048276653a9a1647327d7b4ae74d27bab37ba8c1b WatchSource:0}: Error finding container 138ee7c2a506cdecb11677a048276653a9a1647327d7b4ae74d27bab37ba8c1b: Status 404 returned error can't find the container with id 138ee7c2a506cdecb11677a048276653a9a1647327d7b4ae74d27bab37ba8c1b Nov 03 00:35:14.391911 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.391882 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:14.391980 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.391946 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:14.391980 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.391973 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:14.435314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.435287 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc"] Nov 03 00:35:14.435808 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:35:14.435755 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a413a39_1efa_4c79_8a34_d40431400818.slice/crio-482a1d42e835254e09c000c00559148ab6b2d7051161fe4dd8152e04f76d3542 WatchSource:0}: Error finding container 482a1d42e835254e09c000c00559148ab6b2d7051161fe4dd8152e04f76d3542: Status 404 returned error can't find the container with id 482a1d42e835254e09c000c00559148ab6b2d7051161fe4dd8152e04f76d3542 Nov 03 00:35:14.438263 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.438232 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:14.438340 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.438298 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:14.438340 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.438328 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:14.623334 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.623239 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc"] Nov 03 00:35:14.623456 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:35:14.623366 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd16683_a7b5_4f26_8a6f_2a3160558426.slice/crio-c5712a1541d13033c6a3f3c209dcebe36d8d72d9fecf841939208f49d6076360 WatchSource:0}: Error finding container c5712a1541d13033c6a3f3c209dcebe36d8d72d9fecf841939208f49d6076360: Status 404 returned error can't find the container with id c5712a1541d13033c6a3f3c209dcebe36d8d72d9fecf841939208f49d6076360 Nov 03 00:35:14.625897 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.625868 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:14.625993 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.625928 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:14.625993 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:14.625959 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:15.259800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:15.258230 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" event={"ID":"b90a54d0-dd85-433d-a372-fecd7d8c5106","Type":"ContainerStarted","Data":"b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6"} Nov 03 00:35:15.259800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:15.258314 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" event={"ID":"b90a54d0-dd85-433d-a372-fecd7d8c5106","Type":"ContainerStarted","Data":"138ee7c2a506cdecb11677a048276653a9a1647327d7b4ae74d27bab37ba8c1b"} Nov 03 00:35:15.273798 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:15.266143 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" event={"ID":"2a413a39-1efa-4c79-8a34-d40431400818","Type":"ContainerStarted","Data":"ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997"} Nov 03 00:35:15.273798 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:15.266205 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" event={"ID":"2a413a39-1efa-4c79-8a34-d40431400818","Type":"ContainerStarted","Data":"482a1d42e835254e09c000c00559148ab6b2d7051161fe4dd8152e04f76d3542"} Nov 03 00:35:15.279935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:15.276364 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" event={"ID":"8dd16683-a7b5-4f26-8a6f-2a3160558426","Type":"ContainerStarted","Data":"1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616"} Nov 03 00:35:15.279935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:15.276429 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" event={"ID":"8dd16683-a7b5-4f26-8a6f-2a3160558426","Type":"ContainerStarted","Data":"c5712a1541d13033c6a3f3c209dcebe36d8d72d9fecf841939208f49d6076360"} Nov 03 00:35:15.400801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:15.384614 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" podStartSLOduration=2.38459226 podStartE2EDuration="2.38459226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:15.381531269 +0000 UTC m=+1798.665088947" watchObservedRunningTime="2026-11-03 00:35:15.38459226 +0000 UTC m=+1798.668149937" Nov 03 00:35:15.516291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:15.516198 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" podStartSLOduration=2.516183152 podStartE2EDuration="2.516183152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:15.512642551 +0000 UTC m=+1798.796200224" watchObservedRunningTime="2026-11-03 00:35:15.516183152 +0000 UTC m=+1798.799740825" Nov 03 00:35:15.675000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:15.674924 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" podStartSLOduration=2.674898433 podStartE2EDuration="2.674898433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:15.652381872 +0000 UTC m=+1798.935939547" watchObservedRunningTime="2026-11-03 00:35:15.674898433 +0000 UTC m=+1798.958456107" Nov 03 00:35:16.001997 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:16.001953 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:16.003468 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:16.003433 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:35:16.003612 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:16.003496 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 00:35:16.003690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:16.003641 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:16.005029 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:16.005004 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:35:16.005130 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:16.005055 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" podUID="2a413a39-1efa-4c79-8a34-d40431400818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 00:35:16.253790 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:16.250140 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:16.261886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:16.258532 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:35:16.261886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:16.258609 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 00:35:17.011956 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.011924 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:35:17.012190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.012167 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" podUID="2a413a39-1efa-4c79-8a34-d40431400818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 00:35:17.015169 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.015137 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:35:17.015411 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.015383 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 00:35:17.083803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.067856 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" podUID="fd605632-b5ae-4448-88fc-eb3638a4244b" containerName="payload-processing" containerID="cri-o://1aafcf855384b98cfbbd6526823ed5bf182b2adb963ab7f5deb2d14338fdf85b" gracePeriod=25 Nov 03 00:35:17.180898 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.179944 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" podUID="903d7c7f-1e87-4385-9548-638ff9390eae" containerName="payload-pre-processing" containerID="cri-o://56cea79f85b9532908d8217a0edc42e65af567d07cfaeee592e76a3899e03b07" gracePeriod=25 Nov 03 00:35:17.267864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.245857 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:35:17.267864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.245927 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 00:35:17.732798 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.720977 4304 generic.go:356] "Generic (PLEG): container finished" podID="fd605632-b5ae-4448-88fc-eb3638a4244b" containerID="1aafcf855384b98cfbbd6526823ed5bf182b2adb963ab7f5deb2d14338fdf85b" exitCode=0 Nov 03 00:35:17.732798 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.721153 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" event={"ID":"fd605632-b5ae-4448-88fc-eb3638a4244b","Type":"ContainerDied","Data":"1aafcf855384b98cfbbd6526823ed5bf182b2adb963ab7f5deb2d14338fdf85b"} Nov 03 00:35:17.814870 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.812259 4304 generic.go:356] "Generic (PLEG): container finished" podID="903d7c7f-1e87-4385-9548-638ff9390eae" containerID="56cea79f85b9532908d8217a0edc42e65af567d07cfaeee592e76a3899e03b07" exitCode=0 Nov 03 00:35:17.814870 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:17.813348 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" event={"ID":"903d7c7f-1e87-4385-9548-638ff9390eae","Type":"ContainerDied","Data":"56cea79f85b9532908d8217a0edc42e65af567d07cfaeee592e76a3899e03b07"} Nov 03 00:35:18.118162 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.070966 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:35:18.118162 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.071045 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 00:35:18.118162 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.090930 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:35:18.118162 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.090988 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" podUID="2a413a39-1efa-4c79-8a34-d40431400818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 00:35:18.290801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.290136 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:35:18.290801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.290198 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 00:35:18.416528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.416298 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:35:18.499016 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.498985 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:35:18.511244 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.511212 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd605632-b5ae-4448-88fc-eb3638a4244b-kube-api-access-z68fd\" (UniqueName: \"kubernetes.io/projected/fd605632-b5ae-4448-88fc-eb3638a4244b-kube-api-access-z68fd\") pod \"fd605632-b5ae-4448-88fc-eb3638a4244b\" (UID: \"fd605632-b5ae-4448-88fc-eb3638a4244b\") " Nov 03 00:35:18.511420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.511363 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd605632-b5ae-4448-88fc-eb3638a4244b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd605632-b5ae-4448-88fc-eb3638a4244b-plugins-config-volume\") pod \"fd605632-b5ae-4448-88fc-eb3638a4244b\" (UID: \"fd605632-b5ae-4448-88fc-eb3638a4244b\") " Nov 03 00:35:18.511420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.511394 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903d7c7f-1e87-4385-9548-638ff9390eae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d7c7f-1e87-4385-9548-638ff9390eae-plugins-config-volume\") pod \"903d7c7f-1e87-4385-9548-638ff9390eae\" (UID: \"903d7c7f-1e87-4385-9548-638ff9390eae\") " Nov 03 00:35:18.511532 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.511470 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903d7c7f-1e87-4385-9548-638ff9390eae-kube-api-access-c4qlj\" (UniqueName: \"kubernetes.io/projected/903d7c7f-1e87-4385-9548-638ff9390eae-kube-api-access-c4qlj\") pod \"903d7c7f-1e87-4385-9548-638ff9390eae\" (UID: \"903d7c7f-1e87-4385-9548-638ff9390eae\") " Nov 03 00:35:18.511831 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.511804 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd605632-b5ae-4448-88fc-eb3638a4244b-plugins-config-volume" pod "fd605632-b5ae-4448-88fc-eb3638a4244b" (UID: "fd605632-b5ae-4448-88fc-eb3638a4244b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:18.511935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.511878 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903d7c7f-1e87-4385-9548-638ff9390eae-plugins-config-volume" pod "903d7c7f-1e87-4385-9548-638ff9390eae" (UID: "903d7c7f-1e87-4385-9548-638ff9390eae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:18.511935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.511899 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd605632-b5ae-4448-88fc-eb3638a4244b-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:18.513973 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.513952 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903d7c7f-1e87-4385-9548-638ff9390eae-kube-api-access-c4qlj" pod "903d7c7f-1e87-4385-9548-638ff9390eae" (UID: "903d7c7f-1e87-4385-9548-638ff9390eae"). InnerVolumeSpecName "kube-api-access-c4qlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:18.514436 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.514415 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd605632-b5ae-4448-88fc-eb3638a4244b-kube-api-access-z68fd" pod "fd605632-b5ae-4448-88fc-eb3638a4244b" (UID: "fd605632-b5ae-4448-88fc-eb3638a4244b"). InnerVolumeSpecName "kube-api-access-z68fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:18.613350 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.613317 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d7c7f-1e87-4385-9548-638ff9390eae-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:18.613350 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.613349 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4qlj\" (UniqueName: \"kubernetes.io/projected/903d7c7f-1e87-4385-9548-638ff9390eae-kube-api-access-c4qlj\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:18.613604 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.613364 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68fd\" (UniqueName: \"kubernetes.io/projected/fd605632-b5ae-4448-88fc-eb3638a4244b-kube-api-access-z68fd\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:18.820147 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.820060 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" event={"ID":"903d7c7f-1e87-4385-9548-638ff9390eae","Type":"ContainerDied","Data":"2cf3ecb37f0b02ca63f740c3243cd2ce1055e4309be78bed7fc6a04ddd9359be"} Nov 03 00:35:18.820147 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.820134 4304 scope.go:122] "RemoveContainer" containerID="56cea79f85b9532908d8217a0edc42e65af567d07cfaeee592e76a3899e03b07" Nov 03 00:35:18.902804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.897870 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc" Nov 03 00:35:18.905138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.904300 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" Nov 03 00:35:18.915116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.915091 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6" event={"ID":"fd605632-b5ae-4448-88fc-eb3638a4244b","Type":"ContainerDied","Data":"4245278b471a7f4460f875798b815adbdea5a6d454ceb8dfa4afbe46c1099668"} Nov 03 00:35:18.915248 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.915236 4304 scope.go:122] "RemoveContainer" containerID="1aafcf855384b98cfbbd6526823ed5bf182b2adb963ab7f5deb2d14338fdf85b" Nov 03 00:35:18.997145 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.997109 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:35:18.997272 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:18.997175 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 00:35:19.004179 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:19.003576 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:35:19.004179 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:19.003648 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" podUID="2a413a39-1efa-4c79-8a34-d40431400818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 00:35:19.111536 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:19.111451 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6"] Nov 03 00:35:19.116642 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:19.116613 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-55578b65df-9j9p6"] Nov 03 00:35:19.219454 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:19.219420 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc"] Nov 03 00:35:19.235798 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:19.234102 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-5c9db6b8c9-6m5pc"] Nov 03 00:35:19.250549 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:19.250335 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:35:19.250549 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:19.250396 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 00:35:19.997243 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:19.997207 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:35:19.997854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:19.997822 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 00:35:20.003821 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:20.003802 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:35:20.004007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:20.003989 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" podUID="2a413a39-1efa-4c79-8a34-d40431400818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 00:35:20.247223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:20.247190 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:35:20.247541 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:20.247475 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 00:35:20.858702 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:20.858664 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903d7c7f-1e87-4385-9548-638ff9390eae" path="/var/lib/kubelet/pods/903d7c7f-1e87-4385-9548-638ff9390eae/volumes" Nov 03 00:35:20.859121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:20.859107 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd605632-b5ae-4448-88fc-eb3638a4244b" path="/var/lib/kubelet/pods/fd605632-b5ae-4448-88fc-eb3638a4244b/volumes" Nov 03 00:35:20.997498 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:20.997462 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:35:20.997913 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:20.997510 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 00:35:21.003257 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:21.003230 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:35:21.003370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:21.003276 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" podUID="2a413a39-1efa-4c79-8a34-d40431400818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 00:35:21.246294 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:21.246248 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:35:21.246496 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:21.246317 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 00:35:21.997674 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:21.997636 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:35:21.998218 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:21.997704 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 00:35:22.004052 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:22.004027 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 00:35:22.004196 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:22.004075 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" podUID="2a413a39-1efa-4c79-8a34-d40431400818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 00:35:22.246066 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:22.246021 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:35:22.246268 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:22.246088 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 00:35:23.354581 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:23.354544 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:23.355271 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:23.355209 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:23.356316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:23.356293 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:35:23.620191 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:23.620105 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:23.625110 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:23.624961 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:23.626125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:23.626107 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:35:23.949307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:23.949276 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:23.949819 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:23.949798 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:23.951528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:35:23.951501 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:35:28.656038 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:35:28.656000 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:35:58.655167 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:35:58.655129 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:36:18.746721 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:18.746684 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z"] Nov 03 00:36:18.747248 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:18.746971 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" podUID="d257de99-abfb-45f9-b00f-552faad901e7" containerName="maas-api" containerID="cri-o://aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9" gracePeriod=30 Nov 03 00:36:19.062296 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.062272 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:36:19.097273 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.097246 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d257de99-abfb-45f9-b00f-552faad901e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d257de99-abfb-45f9-b00f-552faad901e7-openshift-service-ca\") pod \"d257de99-abfb-45f9-b00f-552faad901e7\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " Nov 03 00:36:19.097425 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.097406 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-maas-api-tls\") pod \"d257de99-abfb-45f9-b00f-552faad901e7\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " Nov 03 00:36:19.097487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.097468 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-metrics-certs\") pod \"d257de99-abfb-45f9-b00f-552faad901e7\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " Nov 03 00:36:19.097576 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.097561 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d257de99-abfb-45f9-b00f-552faad901e7-kube-api-access-74fps\" (UniqueName: \"kubernetes.io/projected/d257de99-abfb-45f9-b00f-552faad901e7-kube-api-access-74fps\") pod \"d257de99-abfb-45f9-b00f-552faad901e7\" (UID: \"d257de99-abfb-45f9-b00f-552faad901e7\") " Nov 03 00:36:19.097663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.097620 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d257de99-abfb-45f9-b00f-552faad901e7-openshift-service-ca" pod "d257de99-abfb-45f9-b00f-552faad901e7" (UID: "d257de99-abfb-45f9-b00f-552faad901e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:19.097963 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.097940 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d257de99-abfb-45f9-b00f-552faad901e7-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.099640 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.099613 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-metrics-certs" pod "d257de99-abfb-45f9-b00f-552faad901e7" (UID: "d257de99-abfb-45f9-b00f-552faad901e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:36:19.099640 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.099658 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-maas-api-tls" pod "d257de99-abfb-45f9-b00f-552faad901e7" (UID: "d257de99-abfb-45f9-b00f-552faad901e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:36:19.099944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.099914 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d257de99-abfb-45f9-b00f-552faad901e7-kube-api-access-74fps" pod "d257de99-abfb-45f9-b00f-552faad901e7" (UID: "d257de99-abfb-45f9-b00f-552faad901e7"). InnerVolumeSpecName "kube-api-access-74fps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:19.214173 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.200655 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.214441 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.214410 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d257de99-abfb-45f9-b00f-552faad901e7-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.214602 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.214446 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74fps\" (UniqueName: \"kubernetes.io/projected/d257de99-abfb-45f9-b00f-552faad901e7-kube-api-access-74fps\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.221967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.221935 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5"] Nov 03 00:36:19.275687 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.275656 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8"] Nov 03 00:36:19.275898 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.275860 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" podUID="25334bc9-eb19-4379-a2f0-510f579bdeeb" containerName="payload-pre-processing" containerID="cri-o://c30a95ba8d6801e085f081e51df8e256d9e90d1ce56e73a9afe4db8ac4c49e87" gracePeriod=30 Nov 03 00:36:19.400713 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.400683 4304 generic.go:356] "Generic (PLEG): container finished" podID="d257de99-abfb-45f9-b00f-552faad901e7" containerID="aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9" exitCode=0 Nov 03 00:36:19.400893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.400755 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" event={"ID":"d257de99-abfb-45f9-b00f-552faad901e7","Type":"ContainerDied","Data":"aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9"} Nov 03 00:36:19.400893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.400825 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" event={"ID":"d257de99-abfb-45f9-b00f-552faad901e7","Type":"ContainerDied","Data":"950c31bad21c030efcc1a71e1b07707c7cc8acf19db3af1d2d444cca55ee013e"} Nov 03 00:36:19.400893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.400853 4304 scope.go:122] "RemoveContainer" containerID="aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9" Nov 03 00:36:19.400893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.400876 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z" Nov 03 00:36:19.405755 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.405724 4304 generic.go:356] "Generic (PLEG): container finished" podID="25334bc9-eb19-4379-a2f0-510f579bdeeb" containerID="c30a95ba8d6801e085f081e51df8e256d9e90d1ce56e73a9afe4db8ac4c49e87" exitCode=0 Nov 03 00:36:19.405972 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.405948 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" event={"ID":"25334bc9-eb19-4379-a2f0-510f579bdeeb","Type":"ContainerDied","Data":"c30a95ba8d6801e085f081e51df8e256d9e90d1ce56e73a9afe4db8ac4c49e87"} Nov 03 00:36:19.406118 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.406044 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" podUID="e2551088-6824-4725-bc0f-c66120219137" containerName="payload-processing" containerID="cri-o://e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b" gracePeriod=30 Nov 03 00:36:19.421059 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.421037 4304 scope.go:122] "RemoveContainer" containerID="aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9" Nov 03 00:36:19.421706 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:36:19.421683 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9\": container with ID starting with aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9 not found: ID does not exist" containerID="aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9" Nov 03 00:36:19.421815 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.421720 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9"} err="failed to get container status \"aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9\": rpc error: code = NotFound desc = could not find container \"aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9\": container with ID starting with aabf0a5f01748c09bce78a084a0d58535175d20a9478cc3b25742032abb0eea9 not found: ID does not exist" Nov 03 00:36:19.587693 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.587666 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:36:19.620857 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.620832 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25334bc9-eb19-4379-a2f0-510f579bdeeb-kube-api-access-h97sj\" (UniqueName: \"kubernetes.io/projected/25334bc9-eb19-4379-a2f0-510f579bdeeb-kube-api-access-h97sj\") pod \"25334bc9-eb19-4379-a2f0-510f579bdeeb\" (UID: \"25334bc9-eb19-4379-a2f0-510f579bdeeb\") " Nov 03 00:36:19.621025 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.620983 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25334bc9-eb19-4379-a2f0-510f579bdeeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25334bc9-eb19-4379-a2f0-510f579bdeeb-plugins-config-volume\") pod \"25334bc9-eb19-4379-a2f0-510f579bdeeb\" (UID: \"25334bc9-eb19-4379-a2f0-510f579bdeeb\") " Nov 03 00:36:19.621898 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.621871 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25334bc9-eb19-4379-a2f0-510f579bdeeb-plugins-config-volume" pod "25334bc9-eb19-4379-a2f0-510f579bdeeb" (UID: "25334bc9-eb19-4379-a2f0-510f579bdeeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:19.622323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.622304 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25334bc9-eb19-4379-a2f0-510f579bdeeb-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.624605 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.624584 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25334bc9-eb19-4379-a2f0-510f579bdeeb-kube-api-access-h97sj" pod "25334bc9-eb19-4379-a2f0-510f579bdeeb" (UID: "25334bc9-eb19-4379-a2f0-510f579bdeeb"). InnerVolumeSpecName "kube-api-access-h97sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:19.679308 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.679282 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z"] Nov 03 00:36:19.683235 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.683207 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bb5643-7d8c4c9576-k2q7z"] Nov 03 00:36:19.688506 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.688487 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:36:19.723510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.723474 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2551088-6824-4725-bc0f-c66120219137-kube-api-access-b6dtg\" (UniqueName: \"kubernetes.io/projected/e2551088-6824-4725-bc0f-c66120219137-kube-api-access-b6dtg\") pod \"e2551088-6824-4725-bc0f-c66120219137\" (UID: \"e2551088-6824-4725-bc0f-c66120219137\") " Nov 03 00:36:19.723759 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.723524 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2551088-6824-4725-bc0f-c66120219137-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2551088-6824-4725-bc0f-c66120219137-plugins-config-volume\") pod \"e2551088-6824-4725-bc0f-c66120219137\" (UID: \"e2551088-6824-4725-bc0f-c66120219137\") " Nov 03 00:36:19.724136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.724110 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h97sj\" (UniqueName: \"kubernetes.io/projected/25334bc9-eb19-4379-a2f0-510f579bdeeb-kube-api-access-h97sj\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.724697 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.724540 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2551088-6824-4725-bc0f-c66120219137-plugins-config-volume" pod "e2551088-6824-4725-bc0f-c66120219137" (UID: "e2551088-6824-4725-bc0f-c66120219137"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:19.727436 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.727412 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2551088-6824-4725-bc0f-c66120219137-kube-api-access-b6dtg" pod "e2551088-6824-4725-bc0f-c66120219137" (UID: "e2551088-6824-4725-bc0f-c66120219137"). InnerVolumeSpecName "kube-api-access-b6dtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:19.825508 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.825476 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2551088-6824-4725-bc0f-c66120219137-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.825508 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:19.825506 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6dtg\" (UniqueName: \"kubernetes.io/projected/e2551088-6824-4725-bc0f-c66120219137-kube-api-access-b6dtg\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:20.415146 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.415120 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" Nov 03 00:36:20.415313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.415149 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8" event={"ID":"25334bc9-eb19-4379-a2f0-510f579bdeeb","Type":"ContainerDied","Data":"15e822a23178defabe0e88917f0130f5536844d4334c6ce32259b670964e9f23"} Nov 03 00:36:20.415313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.415211 4304 scope.go:122] "RemoveContainer" containerID="c30a95ba8d6801e085f081e51df8e256d9e90d1ce56e73a9afe4db8ac4c49e87" Nov 03 00:36:20.416484 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.416459 4304 generic.go:356] "Generic (PLEG): container finished" podID="e2551088-6824-4725-bc0f-c66120219137" containerID="e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b" exitCode=0 Nov 03 00:36:20.416560 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.416505 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" Nov 03 00:36:20.416560 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.416538 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" event={"ID":"e2551088-6824-4725-bc0f-c66120219137","Type":"ContainerDied","Data":"e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b"} Nov 03 00:36:20.416670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.416611 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5" event={"ID":"e2551088-6824-4725-bc0f-c66120219137","Type":"ContainerDied","Data":"c84d01bfa3f48691de067bedd0dddd72353a04fac1d7bcd7f0481f2f4a5fdffe"} Nov 03 00:36:20.426553 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.426536 4304 scope.go:122] "RemoveContainer" containerID="e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b" Nov 03 00:36:20.438291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.435959 4304 scope.go:122] "RemoveContainer" containerID="e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b" Nov 03 00:36:20.438659 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:36:20.438639 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b\": container with ID starting with e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b not found: ID does not exist" containerID="e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b" Nov 03 00:36:20.438726 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.438668 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b"} err="failed to get container status \"e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b\": rpc error: code = NotFound desc = could not find container \"e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b\": container with ID starting with e5c6464b763c4b82c68bf843e849a37bea5c3c62f1e152e55774dd7568f6f29b not found: ID does not exist" Nov 03 00:36:20.580006 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.579976 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5"] Nov 03 00:36:20.588015 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.587992 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-5556469d46-w7fh5"] Nov 03 00:36:20.698465 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.698375 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8"] Nov 03 00:36:20.701908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.701883 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-5c5c64fd75-86kg8"] Nov 03 00:36:20.829544 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.829511 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25334bc9-eb19-4379-a2f0-510f579bdeeb" path="/var/lib/kubelet/pods/25334bc9-eb19-4379-a2f0-510f579bdeeb/volumes" Nov 03 00:36:20.829969 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.829955 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d257de99-abfb-45f9-b00f-552faad901e7" path="/var/lib/kubelet/pods/d257de99-abfb-45f9-b00f-552faad901e7/volumes" Nov 03 00:36:20.830288 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.830277 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2551088-6824-4725-bc0f-c66120219137" path="/var/lib/kubelet/pods/e2551088-6824-4725-bc0f-c66120219137/volumes" Nov 03 00:36:20.881237 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881203 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c"] Nov 03 00:36:20.881701 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881687 4304 cpu_manager.go:414] "containerMap: removing container" podUID="903d7c7f-1e87-4385-9548-638ff9390eae" containerName="payload-pre-processing" Nov 03 00:36:20.881750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881702 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903d7c7f-1e87-4385-9548-638ff9390eae" containerName="payload-pre-processing" Nov 03 00:36:20.881750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881722 4304 cpu_manager.go:414] "containerMap: removing container" podUID="d257de99-abfb-45f9-b00f-552faad901e7" containerName="maas-api" Nov 03 00:36:20.881750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881730 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d257de99-abfb-45f9-b00f-552faad901e7" containerName="maas-api" Nov 03 00:36:20.881750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881741 4304 cpu_manager.go:414] "containerMap: removing container" podUID="fd605632-b5ae-4448-88fc-eb3638a4244b" containerName="payload-processing" Nov 03 00:36:20.881750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881746 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd605632-b5ae-4448-88fc-eb3638a4244b" containerName="payload-processing" Nov 03 00:36:20.881933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881754 4304 cpu_manager.go:414] "containerMap: removing container" podUID="25334bc9-eb19-4379-a2f0-510f579bdeeb" containerName="payload-pre-processing" Nov 03 00:36:20.881933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881759 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25334bc9-eb19-4379-a2f0-510f579bdeeb" containerName="payload-pre-processing" Nov 03 00:36:20.881933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881768 4304 cpu_manager.go:414] "containerMap: removing container" podUID="e2551088-6824-4725-bc0f-c66120219137" containerName="payload-processing" Nov 03 00:36:20.881933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881786 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2551088-6824-4725-bc0f-c66120219137" containerName="payload-processing" Nov 03 00:36:20.881933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881910 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="903d7c7f-1e87-4385-9548-638ff9390eae" containerName="payload-pre-processing" Nov 03 00:36:20.881933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881919 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d257de99-abfb-45f9-b00f-552faad901e7" containerName="maas-api" Nov 03 00:36:20.881933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881925 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd605632-b5ae-4448-88fc-eb3638a4244b" containerName="payload-processing" Nov 03 00:36:20.881933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881933 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="25334bc9-eb19-4379-a2f0-510f579bdeeb" containerName="payload-pre-processing" Nov 03 00:36:20.882163 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.881939 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2551088-6824-4725-bc0f-c66120219137" containerName="payload-processing" Nov 03 00:36:20.886424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.886410 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:20.969190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.938409 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842da41d-8b5e-4197-9b92-5fa6e4d57562-istiod-ca-cert\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:20.969190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.938555 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-credential-socket\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:20.969190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.938589 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-data\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:20.969190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.938628 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-token\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:20.969190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.938700 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-podinfo\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:20.969190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.938732 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glpqx\" (UniqueName: \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-kube-api-access-glpqx\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:20.969190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.938767 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-socket\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:20.969190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.938807 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-envoy\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:20.969190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.938831 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-certs\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.015072 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.989817 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-011885-openshift-default-dockercfg-r9ffd\"" Nov 03 00:36:21.015072 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:20.994044 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c"] Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.040122 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-credential-socket\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.040181 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-data\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.040233 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-token\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.040319 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-podinfo\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.040353 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glpqx\" (UniqueName: \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-kube-api-access-glpqx\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.040394 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-socket\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.040417 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-envoy\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.040583 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-credential-socket\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.041049 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-certs\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.041132 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842da41d-8b5e-4197-9b92-5fa6e4d57562-istiod-ca-cert\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.042009 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842da41d-8b5e-4197-9b92-5fa6e4d57562-istiod-ca-cert\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.042541 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-socket\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.047455 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-envoy\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.047698 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-certs\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.051807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.048038 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-data\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.063219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.063188 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-podinfo\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.099728 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.099693 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glpqx\" (UniqueName: \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-kube-api-access-glpqx\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.106899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.106871 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-token\") pod \"e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.198886 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.198853 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:21.626174 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.626144 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c"] Nov 03 00:36:21.626652 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:36:21.626619 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842da41d_8b5e_4197_9b92_5fa6e4d57562.slice/crio-f85a6b35e4c1a2306ada6a11c6043e340a3040686bdb9a1f20674d32e2959567 WatchSource:0}: Error finding container f85a6b35e4c1a2306ada6a11c6043e340a3040686bdb9a1f20674d32e2959567: Status 404 returned error can't find the container with id f85a6b35e4c1a2306ada6a11c6043e340a3040686bdb9a1f20674d32e2959567 Nov 03 00:36:21.629052 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.629022 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:21.629123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.629085 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:21.629123 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:21.629114 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:22.435865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:22.435829 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" event={"ID":"842da41d-8b5e-4197-9b92-5fa6e4d57562","Type":"ContainerStarted","Data":"e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f"} Nov 03 00:36:22.436286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:22.435878 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" event={"ID":"842da41d-8b5e-4197-9b92-5fa6e4d57562","Type":"ContainerStarted","Data":"f85a6b35e4c1a2306ada6a11c6043e340a3040686bdb9a1f20674d32e2959567"} Nov 03 00:36:22.554186 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:22.554135 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" podStartSLOduration=2.554121556 podStartE2EDuration="2.554121556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:22.551804852 +0000 UTC m=+1865.835362525" watchObservedRunningTime="2026-11-03 00:36:22.554121556 +0000 UTC m=+1865.837679230" Nov 03 00:36:23.199111 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:23.199079 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:23.204413 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:23.204390 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:23.442458 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:23.442425 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:23.443442 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:23.443427 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:36:28.655584 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:36:28.655541 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:36:28.700206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:28.700176 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld"] Nov 03 00:36:28.700441 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:28.700417 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" containerID="cri-o://b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd" gracePeriod=30 Nov 03 00:36:31.779096 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:31.779064 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9"] Nov 03 00:36:31.779572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:31.779388 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" containerID="cri-o://f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678" gracePeriod=30 Nov 03 00:36:33.973015 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:33.972959 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:36:34.116121 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.116093 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-data\") pod \"32844841-6231-4493-92e4-46a7ff8a67bb\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " Nov 03 00:36:34.116298 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.116167 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32844841-6231-4493-92e4-46a7ff8a67bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32844841-6231-4493-92e4-46a7ff8a67bb-istiod-ca-cert\") pod \"32844841-6231-4493-92e4-46a7ff8a67bb\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " Nov 03 00:36:34.116298 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.116198 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-istio-token\" (UniqueName: \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-istio-token\") pod \"32844841-6231-4493-92e4-46a7ff8a67bb\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " Nov 03 00:36:34.116298 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.116291 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-envoy\") pod \"32844841-6231-4493-92e4-46a7ff8a67bb\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " Nov 03 00:36:34.116424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.116317 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-kube-api-access-r7k8p\" (UniqueName: \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-kube-api-access-r7k8p\") pod \"32844841-6231-4493-92e4-46a7ff8a67bb\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " Nov 03 00:36:34.116424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.116361 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-certs\") pod \"32844841-6231-4493-92e4-46a7ff8a67bb\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " Nov 03 00:36:34.116424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.116387 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-socket\") pod \"32844841-6231-4493-92e4-46a7ff8a67bb\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " Nov 03 00:36:34.116532 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.116449 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-credential-socket\") pod \"32844841-6231-4493-92e4-46a7ff8a67bb\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " Nov 03 00:36:34.116532 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.116474 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32844841-6231-4493-92e4-46a7ff8a67bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32844841-6231-4493-92e4-46a7ff8a67bb-istio-podinfo\") pod \"32844841-6231-4493-92e4-46a7ff8a67bb\" (UID: \"32844841-6231-4493-92e4-46a7ff8a67bb\") " Nov 03 00:36:34.116532 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.116467 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-data" pod "32844841-6231-4493-92e4-46a7ff8a67bb" (UID: "32844841-6231-4493-92e4-46a7ff8a67bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:34.117417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.116800 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.117417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.117023 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-certs" pod "32844841-6231-4493-92e4-46a7ff8a67bb" (UID: "32844841-6231-4493-92e4-46a7ff8a67bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:34.117417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.117153 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-credential-socket" pod "32844841-6231-4493-92e4-46a7ff8a67bb" (UID: "32844841-6231-4493-92e4-46a7ff8a67bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:34.117417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.117062 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-socket" pod "32844841-6231-4493-92e4-46a7ff8a67bb" (UID: "32844841-6231-4493-92e4-46a7ff8a67bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:34.117417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.117367 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32844841-6231-4493-92e4-46a7ff8a67bb-istiod-ca-cert" pod "32844841-6231-4493-92e4-46a7ff8a67bb" (UID: "32844841-6231-4493-92e4-46a7ff8a67bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:34.119806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.119172 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-kube-api-access-r7k8p" pod "32844841-6231-4493-92e4-46a7ff8a67bb" (UID: "32844841-6231-4493-92e4-46a7ff8a67bb"). InnerVolumeSpecName "kube-api-access-r7k8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:34.119806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.119414 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-istio-token" pod "32844841-6231-4493-92e4-46a7ff8a67bb" (UID: "32844841-6231-4493-92e4-46a7ff8a67bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:34.119806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.119557 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-envoy" pod "32844841-6231-4493-92e4-46a7ff8a67bb" (UID: "32844841-6231-4493-92e4-46a7ff8a67bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:34.121118 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.121092 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32844841-6231-4493-92e4-46a7ff8a67bb-istio-podinfo" pod "32844841-6231-4493-92e4-46a7ff8a67bb" (UID: "32844841-6231-4493-92e4-46a7ff8a67bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:36:34.217469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.217438 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.217469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.217466 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32844841-6231-4493-92e4-46a7ff8a67bb-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.217469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.217476 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32844841-6231-4493-92e4-46a7ff8a67bb-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.217469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.217485 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.217744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.217494 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.217744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.217502 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7k8p\" (UniqueName: \"kubernetes.io/projected/32844841-6231-4493-92e4-46a7ff8a67bb-kube-api-access-r7k8p\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.217744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.217512 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.217744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.217520 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32844841-6231-4493-92e4-46a7ff8a67bb-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.517427 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.517389 4304 generic.go:356] "Generic (PLEG): container finished" podID="32844841-6231-4493-92e4-46a7ff8a67bb" containerID="b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd" exitCode=0 Nov 03 00:36:34.517630 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.517462 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" Nov 03 00:36:34.517630 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.517471 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" event={"ID":"32844841-6231-4493-92e4-46a7ff8a67bb","Type":"ContainerDied","Data":"b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd"} Nov 03 00:36:34.517630 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.517551 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld" event={"ID":"32844841-6231-4493-92e4-46a7ff8a67bb","Type":"ContainerDied","Data":"69b391eaa64f13500a4780a8690053cbdc2ed2f12414e3883d884eadf93ea7cb"} Nov 03 00:36:34.517630 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.517573 4304 scope.go:122] "RemoveContainer" containerID="b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd" Nov 03 00:36:34.528019 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.527998 4304 scope.go:122] "RemoveContainer" containerID="b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd" Nov 03 00:36:34.528319 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:36:34.528300 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd\": container with ID starting with b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd not found: ID does not exist" containerID="b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd" Nov 03 00:36:34.528364 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.528327 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd"} err="failed to get container status \"b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd\": rpc error: code = NotFound desc = could not find container \"b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd\": container with ID starting with b3f35331b4edffead12ce0c84d4784c3690490ad4355f8891bfd629485de89bd not found: ID does not exist" Nov 03 00:36:34.688934 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.688897 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld"] Nov 03 00:36:34.733594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.733566 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-vj6ld"] Nov 03 00:36:34.829406 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:34.829327 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" path="/var/lib/kubelet/pods/32844841-6231-4493-92e4-46a7ff8a67bb/volumes" Nov 03 00:36:37.051558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.051506 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:36:37.146425 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.146395 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-podinfo\") pod \"c2d8924d-ef4a-4891-9492-b257cbdc7647\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " Nov 03 00:36:37.146581 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.146431 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-kube-api-access-lkfjj\" (UniqueName: \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-kube-api-access-lkfjj\") pod \"c2d8924d-ef4a-4891-9492-b257cbdc7647\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " Nov 03 00:36:37.146581 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.146472 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-credential-socket\") pod \"c2d8924d-ef4a-4891-9492-b257cbdc7647\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " Nov 03 00:36:37.146581 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.146505 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-socket\") pod \"c2d8924d-ef4a-4891-9492-b257cbdc7647\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " Nov 03 00:36:37.146581 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.146553 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-token\" (UniqueName: \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-token\") pod \"c2d8924d-ef4a-4891-9492-b257cbdc7647\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " Nov 03 00:36:37.146763 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.146583 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-envoy\") pod \"c2d8924d-ef4a-4891-9492-b257cbdc7647\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " Nov 03 00:36:37.146763 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.146672 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d8924d-ef4a-4891-9492-b257cbdc7647-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d8924d-ef4a-4891-9492-b257cbdc7647-istiod-ca-cert\") pod \"c2d8924d-ef4a-4891-9492-b257cbdc7647\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " Nov 03 00:36:37.146763 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.146713 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-data\") pod \"c2d8924d-ef4a-4891-9492-b257cbdc7647\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " Nov 03 00:36:37.146763 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.146744 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-certs\") pod \"c2d8924d-ef4a-4891-9492-b257cbdc7647\" (UID: \"c2d8924d-ef4a-4891-9492-b257cbdc7647\") " Nov 03 00:36:37.147015 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.146884 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-socket" pod "c2d8924d-ef4a-4891-9492-b257cbdc7647" (UID: "c2d8924d-ef4a-4891-9492-b257cbdc7647"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:37.147015 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.146881 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-credential-socket" pod "c2d8924d-ef4a-4891-9492-b257cbdc7647" (UID: "c2d8924d-ef4a-4891-9492-b257cbdc7647"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:37.147127 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.147099 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-data" pod "c2d8924d-ef4a-4891-9492-b257cbdc7647" (UID: "c2d8924d-ef4a-4891-9492-b257cbdc7647"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:37.147186 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.147167 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d8924d-ef4a-4891-9492-b257cbdc7647-istiod-ca-cert" pod "c2d8924d-ef4a-4891-9492-b257cbdc7647" (UID: "c2d8924d-ef4a-4891-9492-b257cbdc7647"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:37.147280 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.147253 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-certs" pod "c2d8924d-ef4a-4891-9492-b257cbdc7647" (UID: "c2d8924d-ef4a-4891-9492-b257cbdc7647"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:37.147368 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.147350 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:37.147430 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.147373 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:37.147430 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.147387 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d8924d-ef4a-4891-9492-b257cbdc7647-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:37.147430 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.147396 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:37.148983 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.148956 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-kube-api-access-lkfjj" pod "c2d8924d-ef4a-4891-9492-b257cbdc7647" (UID: "c2d8924d-ef4a-4891-9492-b257cbdc7647"). InnerVolumeSpecName "kube-api-access-lkfjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:37.149126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.149098 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-envoy" pod "c2d8924d-ef4a-4891-9492-b257cbdc7647" (UID: "c2d8924d-ef4a-4891-9492-b257cbdc7647"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:37.149317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.149295 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-podinfo" pod "c2d8924d-ef4a-4891-9492-b257cbdc7647" (UID: "c2d8924d-ef4a-4891-9492-b257cbdc7647"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:36:37.149385 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.149312 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-token" pod "c2d8924d-ef4a-4891-9492-b257cbdc7647" (UID: "c2d8924d-ef4a-4891-9492-b257cbdc7647"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:37.248534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.248497 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:37.248534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.248524 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkfjj\" (UniqueName: \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-kube-api-access-lkfjj\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:37.248534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.248536 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:37.248534 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.248545 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:37.248930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.248554 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d8924d-ef4a-4891-9492-b257cbdc7647-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:37.539468 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.539434 4304 generic.go:356] "Generic (PLEG): container finished" podID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerID="f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678" exitCode=0 Nov 03 00:36:37.539646 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.539488 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" Nov 03 00:36:37.539646 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.539490 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" event={"ID":"c2d8924d-ef4a-4891-9492-b257cbdc7647","Type":"ContainerDied","Data":"f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678"} Nov 03 00:36:37.539646 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.539616 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9" event={"ID":"c2d8924d-ef4a-4891-9492-b257cbdc7647","Type":"ContainerDied","Data":"6a95e58b92461284530ffc399a37a1b960707c2bc831a5ac50617bb1928ea14e"} Nov 03 00:36:37.539760 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.539647 4304 scope.go:122] "RemoveContainer" containerID="f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678" Nov 03 00:36:37.550541 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.550522 4304 scope.go:122] "RemoveContainer" containerID="05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759" Nov 03 00:36:37.559509 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.559490 4304 scope.go:122] "RemoveContainer" containerID="f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678" Nov 03 00:36:37.559798 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:36:37.559756 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678\": container with ID starting with f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678 not found: ID does not exist" containerID="f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678" Nov 03 00:36:37.559876 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.559808 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678"} err="failed to get container status \"f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678\": rpc error: code = NotFound desc = could not find container \"f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678\": container with ID starting with f8558e593557a54722ddd4eb403316fa1b1a27434396d959f4efe7f100fb9678 not found: ID does not exist" Nov 03 00:36:37.559876 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.559827 4304 scope.go:122] "RemoveContainer" containerID="05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759" Nov 03 00:36:37.560104 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:36:37.560087 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759\": container with ID starting with 05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759 not found: ID does not exist" containerID="05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759" Nov 03 00:36:37.560155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.560110 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759"} err="failed to get container status \"05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759\": rpc error: code = NotFound desc = could not find container \"05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759\": container with ID starting with 05bf419e9a6b92b8e821a9de4848ece0e6a5b04982d915b15687249b8a121759 not found: ID does not exist" Nov 03 00:36:37.705383 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.705354 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9"] Nov 03 00:36:37.755219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:37.755190 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bb5643-openshift-default-bb5fdc7cf-hx8z9"] Nov 03 00:36:38.829147 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:38.829114 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" path="/var/lib/kubelet/pods/c2d8924d-ef4a-4891-9492-b257cbdc7647/volumes" Nov 03 00:36:49.547671 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.547631 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6"] Nov 03 00:36:49.549964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.548232 4304 cpu_manager.go:414] "containerMap: removing container" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" Nov 03 00:36:49.549964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.548251 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" Nov 03 00:36:49.549964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.548280 4304 cpu_manager.go:414] "containerMap: removing container" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" Nov 03 00:36:49.549964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.548288 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" Nov 03 00:36:49.549964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.548318 4304 cpu_manager.go:414] "containerMap: removing container" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" Nov 03 00:36:49.549964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.548326 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" Nov 03 00:36:49.549964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.548454 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" Nov 03 00:36:49.549964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.548470 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="32844841-6231-4493-92e4-46a7ff8a67bb" containerName="istio-proxy" Nov 03 00:36:49.551570 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.551555 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.642351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.628106 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-socket\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.642351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.628184 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-data\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.642351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.628234 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-token\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.642351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.628282 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-envoy\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.642351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.628326 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3765fa10-9d25-4251-b5c4-485fa784a0e1-istiod-ca-cert\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.642351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.628350 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvgvp\" (UniqueName: \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-kube-api-access-qvgvp\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.642351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.628386 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-certs\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.642351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.628414 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-credential-socket\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.642351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.628436 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-podinfo\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.677366 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.677325 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-75dfee-openshift-default-dockercfg-gxl9m\"" Nov 03 00:36:49.682957 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.682925 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6"] Nov 03 00:36:49.729615 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.729563 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-socket\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.729832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.729660 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-data\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.729832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.729716 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-token\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.729832 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.729822 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-envoy\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.730029 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.729878 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3765fa10-9d25-4251-b5c4-485fa784a0e1-istiod-ca-cert\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.730029 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.729903 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvgvp\" (UniqueName: \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-kube-api-access-qvgvp\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.730029 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.729948 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-certs\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.730029 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.729981 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-credential-socket\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.730029 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.730006 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-podinfo\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.731207 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.731167 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-data\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.731559 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.731530 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3765fa10-9d25-4251-b5c4-485fa784a0e1-istiod-ca-cert\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.731863 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.731840 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-certs\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.732069 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.732046 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-socket\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.732207 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.732098 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-credential-socket\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.738831 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.737603 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-envoy\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.738831 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.738042 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-podinfo\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.786746 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.786706 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvgvp\" (UniqueName: \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-kube-api-access-qvgvp\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.787737 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.787710 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-token\") pod \"e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:49.864873 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:49.864807 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:50.289092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:50.289063 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6"] Nov 03 00:36:50.290065 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:36:50.290037 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3765fa10_9d25_4251_b5c4_485fa784a0e1.slice/crio-f7c5b0b19676aab73990e6dc991b338bae3d1ad9e56641593667835926fdc18e WatchSource:0}: Error finding container f7c5b0b19676aab73990e6dc991b338bae3d1ad9e56641593667835926fdc18e: Status 404 returned error can't find the container with id f7c5b0b19676aab73990e6dc991b338bae3d1ad9e56641593667835926fdc18e Nov 03 00:36:50.292537 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:50.292505 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:50.292613 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:50.292571 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:50.292613 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:50.292597 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:50.654272 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:50.654233 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" event={"ID":"3765fa10-9d25-4251-b5c4-485fa784a0e1","Type":"ContainerStarted","Data":"c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b"} Nov 03 00:36:50.654636 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:50.654283 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" event={"ID":"3765fa10-9d25-4251-b5c4-485fa784a0e1","Type":"ContainerStarted","Data":"f7c5b0b19676aab73990e6dc991b338bae3d1ad9e56641593667835926fdc18e"} Nov 03 00:36:50.772569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:50.772506 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" podStartSLOduration=1.772486365 podStartE2EDuration="1.772486365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:50.769978674 +0000 UTC m=+1894.053536350" watchObservedRunningTime="2026-11-03 00:36:50.772486365 +0000 UTC m=+1894.056044039" Nov 03 00:36:50.865906 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:50.865879 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:51.870550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:51.870523 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:52.667978 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:52.667952 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:52.668890 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:36:52.668872 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:36:58.655279 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:36:58.655237 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:37:03.865479 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:03.865441 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4"] Nov 03 00:37:03.865977 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:03.865681 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" podUID="df822e75-e797-4eaf-af86-88f3af76deed" containerName="payload-processing" containerID="cri-o://b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701" gracePeriod=30 Nov 03 00:37:03.925071 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:03.925042 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x"] Nov 03 00:37:03.925236 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:03.925218 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" podUID="d9cb7dee-1a14-48c2-bba3-19af10d4c28b" containerName="payload-pre-processing" containerID="cri-o://45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789" gracePeriod=30 Nov 03 00:37:04.215392 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.215367 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:37:04.232048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.232029 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:37:04.311595 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.311561 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df822e75-e797-4eaf-af86-88f3af76deed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df822e75-e797-4eaf-af86-88f3af76deed-plugins-config-volume\") pod \"df822e75-e797-4eaf-af86-88f3af76deed\" (UID: \"df822e75-e797-4eaf-af86-88f3af76deed\") " Nov 03 00:37:04.311744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.311665 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df822e75-e797-4eaf-af86-88f3af76deed-kube-api-access-m42gx\" (UniqueName: \"kubernetes.io/projected/df822e75-e797-4eaf-af86-88f3af76deed-kube-api-access-m42gx\") pod \"df822e75-e797-4eaf-af86-88f3af76deed\" (UID: \"df822e75-e797-4eaf-af86-88f3af76deed\") " Nov 03 00:37:04.311744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.311737 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-plugins-config-volume\") pod \"d9cb7dee-1a14-48c2-bba3-19af10d4c28b\" (UID: \"d9cb7dee-1a14-48c2-bba3-19af10d4c28b\") " Nov 03 00:37:04.311875 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.311858 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-kube-api-access-2mrtp\" (UniqueName: \"kubernetes.io/projected/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-kube-api-access-2mrtp\") pod \"d9cb7dee-1a14-48c2-bba3-19af10d4c28b\" (UID: \"d9cb7dee-1a14-48c2-bba3-19af10d4c28b\") " Nov 03 00:37:04.312027 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.311999 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df822e75-e797-4eaf-af86-88f3af76deed-plugins-config-volume" pod "df822e75-e797-4eaf-af86-88f3af76deed" (UID: "df822e75-e797-4eaf-af86-88f3af76deed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:37:04.312102 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.312084 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-plugins-config-volume" pod "d9cb7dee-1a14-48c2-bba3-19af10d4c28b" (UID: "d9cb7dee-1a14-48c2-bba3-19af10d4c28b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:37:04.312364 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.312344 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:04.312480 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.312366 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df822e75-e797-4eaf-af86-88f3af76deed-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:04.314080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.314058 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df822e75-e797-4eaf-af86-88f3af76deed-kube-api-access-m42gx" pod "df822e75-e797-4eaf-af86-88f3af76deed" (UID: "df822e75-e797-4eaf-af86-88f3af76deed"). InnerVolumeSpecName "kube-api-access-m42gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:04.314169 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.314110 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-kube-api-access-2mrtp" pod "d9cb7dee-1a14-48c2-bba3-19af10d4c28b" (UID: "d9cb7dee-1a14-48c2-bba3-19af10d4c28b"). InnerVolumeSpecName "kube-api-access-2mrtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:04.416674 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.416635 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mrtp\" (UniqueName: \"kubernetes.io/projected/d9cb7dee-1a14-48c2-bba3-19af10d4c28b-kube-api-access-2mrtp\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:04.416674 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.416675 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42gx\" (UniqueName: \"kubernetes.io/projected/df822e75-e797-4eaf-af86-88f3af76deed-kube-api-access-m42gx\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:04.748002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.747902 4304 generic.go:356] "Generic (PLEG): container finished" podID="df822e75-e797-4eaf-af86-88f3af76deed" containerID="b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701" exitCode=0 Nov 03 00:37:04.748002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.747947 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" Nov 03 00:37:04.748002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.747994 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" event={"ID":"df822e75-e797-4eaf-af86-88f3af76deed","Type":"ContainerDied","Data":"b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701"} Nov 03 00:37:04.748417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.748042 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4" event={"ID":"df822e75-e797-4eaf-af86-88f3af76deed","Type":"ContainerDied","Data":"e880842a7776541683d68fbf1efc9815bb62e3d3d9ada2a91bfd30b0704d0884"} Nov 03 00:37:04.748417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.748064 4304 scope.go:122] "RemoveContainer" containerID="b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701" Nov 03 00:37:04.749653 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.749619 4304 generic.go:356] "Generic (PLEG): container finished" podID="d9cb7dee-1a14-48c2-bba3-19af10d4c28b" containerID="45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789" exitCode=0 Nov 03 00:37:04.749807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.749650 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" event={"ID":"d9cb7dee-1a14-48c2-bba3-19af10d4c28b","Type":"ContainerDied","Data":"45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789"} Nov 03 00:37:04.749807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.749675 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" Nov 03 00:37:04.749807 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.749691 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x" event={"ID":"d9cb7dee-1a14-48c2-bba3-19af10d4c28b","Type":"ContainerDied","Data":"9f088ff7dc0148d710be7e0e5438f1d76de44ed30920f112400cde0db51d04e5"} Nov 03 00:37:04.759603 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.759587 4304 scope.go:122] "RemoveContainer" containerID="b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701" Nov 03 00:37:04.759962 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:37:04.759940 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701\": container with ID starting with b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701 not found: ID does not exist" containerID="b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701" Nov 03 00:37:04.760058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.759969 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701"} err="failed to get container status \"b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701\": rpc error: code = NotFound desc = could not find container \"b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701\": container with ID starting with b341b58983c8bd5cbc458baa7df9412f0466f0be4828bb34e24a6dab254e3701 not found: ID does not exist" Nov 03 00:37:04.760058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.759984 4304 scope.go:122] "RemoveContainer" containerID="45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789" Nov 03 00:37:04.768645 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.768629 4304 scope.go:122] "RemoveContainer" containerID="45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789" Nov 03 00:37:04.768931 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:37:04.768913 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789\": container with ID starting with 45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789 not found: ID does not exist" containerID="45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789" Nov 03 00:37:04.768986 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.768938 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789"} err="failed to get container status \"45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789\": rpc error: code = NotFound desc = could not find container \"45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789\": container with ID starting with 45cfb2a828ceed1fb0aff5dd4998b7f11316f4b5d11b4ad3d59ed27314d18789 not found: ID does not exist" Nov 03 00:37:04.925304 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.922459 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x"] Nov 03 00:37:04.926005 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:04.925982 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-84d796c7d7-fkr7x"] Nov 03 00:37:05.028845 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:05.028735 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4"] Nov 03 00:37:05.034446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:05.034422 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3df441-66fd669846-nq6q4"] Nov 03 00:37:06.828708 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:06.828678 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9cb7dee-1a14-48c2-bba3-19af10d4c28b" path="/var/lib/kubelet/pods/d9cb7dee-1a14-48c2-bba3-19af10d4c28b/volumes" Nov 03 00:37:06.829162 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:06.829101 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df822e75-e797-4eaf-af86-88f3af76deed" path="/var/lib/kubelet/pods/df822e75-e797-4eaf-af86-88f3af76deed/volumes" Nov 03 00:37:14.691261 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.691217 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7"] Nov 03 00:37:14.691912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.691691 4304 cpu_manager.go:414] "containerMap: removing container" podUID="d9cb7dee-1a14-48c2-bba3-19af10d4c28b" containerName="payload-pre-processing" Nov 03 00:37:14.691912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.691703 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9cb7dee-1a14-48c2-bba3-19af10d4c28b" containerName="payload-pre-processing" Nov 03 00:37:14.691912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.691745 4304 cpu_manager.go:414] "containerMap: removing container" podUID="df822e75-e797-4eaf-af86-88f3af76deed" containerName="payload-processing" Nov 03 00:37:14.691912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.691750 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df822e75-e797-4eaf-af86-88f3af76deed" containerName="payload-processing" Nov 03 00:37:14.691912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.691852 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="df822e75-e797-4eaf-af86-88f3af76deed" containerName="payload-processing" Nov 03 00:37:14.691912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.691860 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9cb7dee-1a14-48c2-bba3-19af10d4c28b" containerName="payload-pre-processing" Nov 03 00:37:14.691912 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.691874 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d8924d-ef4a-4891-9492-b257cbdc7647" containerName="istio-proxy" Nov 03 00:37:14.695059 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.695030 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.809585 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.807329 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7"] Nov 03 00:37:14.818823 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.810213 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-0db2fb-openshift-default-dockercfg-vzc8h\"" Nov 03 00:37:14.853927 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.853575 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-data\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.853927 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.853766 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-socket\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.853927 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.853828 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-envoy\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.853927 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.853853 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-token\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.853927 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.853900 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvbg\" (UniqueName: \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-kube-api-access-fzvbg\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.853927 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.853935 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-podinfo\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.854442 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.854000 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-certs\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.854442 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.854051 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-credential-socket\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.854442 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.854102 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077ba44b-d69e-42f4-8511-b730c8c4855a-istiod-ca-cert\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.955357 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.955265 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-credential-socket\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.955357 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.955342 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077ba44b-d69e-42f4-8511-b730c8c4855a-istiod-ca-cert\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.955583 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.955406 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-data\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.955583 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.955573 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-socket\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.955693 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.955608 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-envoy\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.955693 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.955631 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-token\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.955693 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.955670 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvbg\" (UniqueName: \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-kube-api-access-fzvbg\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.955880 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.955706 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-podinfo\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.955880 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.955711 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-credential-socket\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.955880 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.955805 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-certs\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.956101 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.956077 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-socket\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.956154 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.956127 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-certs\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.956366 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.956341 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-data\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.957033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.957007 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077ba44b-d69e-42f4-8511-b730c8c4855a-istiod-ca-cert\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.959305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.959279 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-podinfo\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:14.959619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:14.959598 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-envoy\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:15.015140 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.015097 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-token\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:15.015312 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.015190 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvbg\" (UniqueName: \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-kube-api-access-fzvbg\") pod \"e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:15.312885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.312760 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:15.433933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.432621 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz"] Nov 03 00:37:15.439047 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.439019 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.492435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.492394 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz"] Nov 03 00:37:15.492692 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.492670 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-011885-serving-cert\"" Nov 03 00:37:15.569035 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.563688 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-metrics-certs\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.569035 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.563742 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.569035 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.563872 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e549a85b-87a0-495f-923c-3e817744f603-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.569035 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.563975 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt4lg\" (UniqueName: \"kubernetes.io/projected/e549a85b-87a0-495f-923c-3e817744f603-kube-api-access-bt4lg\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.671801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.666486 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-metrics-certs\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.671801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.666543 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.671801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.666646 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e549a85b-87a0-495f-923c-3e817744f603-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.671801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.666748 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt4lg\" (UniqueName: \"kubernetes.io/projected/e549a85b-87a0-495f-923c-3e817744f603-kube-api-access-bt4lg\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.695800 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:37:15.692936 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-serving-cert: secret "maas-api-e2e-ipp-b-011885-serving-cert" not found Nov 03 00:37:15.695800 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:37:15.693029 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls podName:e549a85b-87a0-495f-923c-3e817744f603 nodeName:}" failed. No retries permitted until 2026-11-03 00:37:16.193004788 +0000 UTC m=+1919.476562444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls") pod "maas-api-e2e-ipp-b-011885-7985786468-z4rxz" (UID: "e549a85b-87a0-495f-923c-3e817744f603") : secret "maas-api-e2e-ipp-b-011885-serving-cert" not found Nov 03 00:37:15.695800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.694080 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e549a85b-87a0-495f-923c-3e817744f603-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.701371 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.701344 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-metrics-certs\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.757801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.744640 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt4lg\" (UniqueName: \"kubernetes.io/projected/e549a85b-87a0-495f-923c-3e817744f603-kube-api-access-bt4lg\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:15.816812 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.816754 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7"] Nov 03 00:37:15.880835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.879870 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:37:15.880835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.879966 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:37:15.880835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.880013 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:37:15.907639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:15.907607 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" event={"ID":"077ba44b-d69e-42f4-8511-b730c8c4855a","Type":"ContainerStarted","Data":"83c0d04187f6344e4aea3d592a81f08143fe899b687e28a78961c0e127573ad0"} Nov 03 00:37:16.217552 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:16.217511 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:16.217745 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:37:16.217674 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-serving-cert: secret "maas-api-e2e-ipp-b-011885-serving-cert" not found Nov 03 00:37:16.217820 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:37:16.217752 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls podName:e549a85b-87a0-495f-923c-3e817744f603 nodeName:}" failed. No retries permitted until 2026-11-03 00:37:17.217731817 +0000 UTC m=+1920.501289483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls") pod "maas-api-e2e-ipp-b-011885-7985786468-z4rxz" (UID: "e549a85b-87a0-495f-923c-3e817744f603") : secret "maas-api-e2e-ipp-b-011885-serving-cert" not found Nov 03 00:37:16.934800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:16.921093 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" event={"ID":"077ba44b-d69e-42f4-8511-b730c8c4855a","Type":"ContainerStarted","Data":"2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd"} Nov 03 00:37:17.065798 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:17.062848 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" podStartSLOduration=3.062828004 podStartE2EDuration="3.062828004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:17.060359761 +0000 UTC m=+1920.343917437" watchObservedRunningTime="2026-11-03 00:37:17.062828004 +0000 UTC m=+1920.346385682" Nov 03 00:37:17.273643 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:17.273552 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:17.297795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:17.297397 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls\") pod \"maas-api-e2e-ipp-b-011885-7985786468-z4rxz\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:17.313277 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:17.313247 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:17.319440 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:17.319421 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:17.557475 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:17.557385 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:17.928410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:17.928380 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:17.929469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:17.929446 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:37:17.933611 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:17.933587 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz"] Nov 03 00:37:17.934497 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:37:17.934471 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode549a85b_87a0_495f_923c_3e817744f603.slice/crio-c03123769b1df4175f95db2badc47895f664c4b18522854ecb5592f89805e468 WatchSource:0}: Error finding container c03123769b1df4175f95db2badc47895f664c4b18522854ecb5592f89805e468: Status 404 returned error can't find the container with id c03123769b1df4175f95db2badc47895f664c4b18522854ecb5592f89805e468 Nov 03 00:37:17.936021 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:17.936002 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:37:18.935643 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:18.935613 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" event={"ID":"e549a85b-87a0-495f-923c-3e817744f603","Type":"ContainerStarted","Data":"b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55"} Nov 03 00:37:18.935753 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:18.935667 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" event={"ID":"e549a85b-87a0-495f-923c-3e817744f603","Type":"ContainerStarted","Data":"c03123769b1df4175f95db2badc47895f664c4b18522854ecb5592f89805e468"} Nov 03 00:37:18.935753 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:18.935705 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:19.087382 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:19.087314 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" podStartSLOduration=3.176285228 podStartE2EDuration="4.087299107s" totalImagesPullingTime="911.013879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:19.086731923 +0000 UTC m=+1922.370289607" watchObservedRunningTime="2026-11-03 00:37:19.087299107 +0000 UTC m=+1922.370856780" Nov 03 00:37:24.950575 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:24.950545 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:37:28.655901 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:37:28.655860 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:37:51.133601 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:51.133512 4304 generic.go:356] "Generic (PLEG): container finished" podID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerID="e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f" exitCode=0 Nov 03 00:37:51.133601 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:51.133585 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" event={"ID":"842da41d-8b5e-4197-9b92-5fa6e4d57562","Type":"ContainerDied","Data":"e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f"} Nov 03 00:37:51.134353 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:51.134020 4304 scope.go:122] "RemoveContainer" containerID="e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f" Nov 03 00:37:51.135480 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:51.135449 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:37:51.135552 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:51.135522 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:37:51.135593 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:51.135552 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:37:52.152434 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:52.152396 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" event={"ID":"842da41d-8b5e-4197-9b92-5fa6e4d57562","Type":"ContainerStarted","Data":"623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66"} Nov 03 00:37:52.199701 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:52.199676 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:37:52.200626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:52.200595 4304 patch_prober.go:29] interesting pod/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 00:37:52.200688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:52.200647 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 00:37:53.199298 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:53.199256 4304 patch_prober.go:29] interesting pod/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 00:37:53.199737 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:53.199315 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 00:37:53.442800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:53.442344 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:37:54.199847 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:54.199813 4304 patch_prober.go:29] interesting pod/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 00:37:54.200245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:54.199866 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 00:37:55.199149 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:55.199114 4304 patch_prober.go:29] interesting pod/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 00:37:55.199327 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:55.199165 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 00:37:56.199535 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:56.199503 4304 patch_prober.go:29] interesting pod/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 00:37:56.199956 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:56.199572 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 00:37:57.219608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:57.219574 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:37:57.222907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:37:57.222886 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:37:58.654139 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:37:58.654102 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:38:15.333823 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:15.333753 4304 generic.go:356] "Generic (PLEG): container finished" podID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerID="c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b" exitCode=0 Nov 03 00:38:15.334297 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:15.333829 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" event={"ID":"3765fa10-9d25-4251-b5c4-485fa784a0e1","Type":"ContainerDied","Data":"c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b"} Nov 03 00:38:15.334297 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:15.334252 4304 scope.go:122] "RemoveContainer" containerID="c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b" Nov 03 00:38:15.335732 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:15.335707 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:15.335834 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:15.335819 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:15.335877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:15.335861 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:16.352881 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:16.352842 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" event={"ID":"3765fa10-9d25-4251-b5c4-485fa784a0e1","Type":"ContainerStarted","Data":"19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea"} Nov 03 00:38:16.865510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:16.865481 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:38:16.865704 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:16.865685 4304 patch_prober.go:29] interesting pod/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:38:16.865746 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:16.865728 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 00:38:17.865709 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:17.865666 4304 patch_prober.go:29] interesting pod/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:38:17.866144 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:17.865751 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 00:38:18.865694 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:18.865663 4304 patch_prober.go:29] interesting pod/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:38:18.865981 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:18.865725 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 00:38:19.865590 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:19.865558 4304 patch_prober.go:29] interesting pod/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:38:19.865768 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:19.865611 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 00:38:20.865849 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:20.865817 4304 patch_prober.go:29] interesting pod/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:38:20.866229 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:20.865864 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 00:38:21.869533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:21.869507 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:38:21.870024 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:21.869806 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:38:21.870602 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:21.870583 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:38:26.518375 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.518333 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2"] Nov 03 00:38:26.521660 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.521638 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:38:26.575493 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.575457 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjgt\" (UniqueName: \"kubernetes.io/projected/0692bf41-be30-48ae-9a89-8e0989352a36-kube-api-access-vpjgt\") pod \"payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2\" (UID: \"0692bf41-be30-48ae-9a89-8e0989352a36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:38:26.577651 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.577620 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0692bf41-be30-48ae-9a89-8e0989352a36-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2\" (UID: \"0692bf41-be30-48ae-9a89-8e0989352a36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:38:26.586917 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.586886 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2"] Nov 03 00:38:26.661157 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.661109 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ff10d6\"" Nov 03 00:38:26.662327 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.661910 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ff10d6-dockercfg-wdvpv\"" Nov 03 00:38:26.679490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.679446 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjgt\" (UniqueName: \"kubernetes.io/projected/0692bf41-be30-48ae-9a89-8e0989352a36-kube-api-access-vpjgt\") pod \"payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2\" (UID: \"0692bf41-be30-48ae-9a89-8e0989352a36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:38:26.679700 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.679568 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0692bf41-be30-48ae-9a89-8e0989352a36-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2\" (UID: \"0692bf41-be30-48ae-9a89-8e0989352a36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:38:26.680558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.680532 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0692bf41-be30-48ae-9a89-8e0989352a36-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2\" (UID: \"0692bf41-be30-48ae-9a89-8e0989352a36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:38:26.736866 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.736837 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjgt\" (UniqueName: \"kubernetes.io/projected/0692bf41-be30-48ae-9a89-8e0989352a36-kube-api-access-vpjgt\") pod \"payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2\" (UID: \"0692bf41-be30-48ae-9a89-8e0989352a36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:38:26.834626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:26.834551 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:38:27.207300 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:27.207271 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2"] Nov 03 00:38:27.420667 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:27.420634 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" event={"ID":"0692bf41-be30-48ae-9a89-8e0989352a36","Type":"ContainerStarted","Data":"ed901658cbb61439ad274b3c2ff926e9dfc8ee593f4ecd647a45e483a1b0a18e"} Nov 03 00:38:27.420865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:27.420683 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" event={"ID":"0692bf41-be30-48ae-9a89-8e0989352a36","Type":"ContainerStarted","Data":"df8cbcc2a92062d1d15dd69967e9f0477f19ffefd6663210d779ece793d5779b"} Nov 03 00:38:27.420865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:27.420719 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:38:27.543795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:27.543673 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" podStartSLOduration=1.5436551299999999 podStartE2EDuration="1.54365513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:27.541065654 +0000 UTC m=+1990.824623341" watchObservedRunningTime="2026-11-03 00:38:27.54365513 +0000 UTC m=+1990.827212806" Nov 03 00:38:28.673704 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:38:28.673663 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:38:38.429732 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:38.429703 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:38:42.361762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.361725 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx"] Nov 03 00:38:42.365321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.365304 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:42.468616 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.468581 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d61eae-serving-cert\"" Nov 03 00:38:42.471905 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.471874 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx"] Nov 03 00:38:42.490462 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.490428 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:42.490574 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.490552 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-metrics-certs\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:42.490634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.490615 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77751960-fc94-4513-a20c-c76a475834f0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:42.490723 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.490710 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg8wl\" (UniqueName: \"kubernetes.io/projected/77751960-fc94-4513-a20c-c76a475834f0-kube-api-access-vg8wl\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:42.592107 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.592073 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:42.592312 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.592177 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-metrics-certs\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:42.592312 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:38:42.592224 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-serving-cert: secret "maas-api-e2e-praxis-a-d61eae-serving-cert" not found Nov 03 00:38:42.592312 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.592232 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77751960-fc94-4513-a20c-c76a475834f0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:42.592312 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:38:42.592284 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-maas-api-tls podName:77751960-fc94-4513-a20c-c76a475834f0 nodeName:}" failed. No retries permitted until 2026-11-03 00:38:43.092268671 +0000 UTC m=+2006.375826324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-maas-api-tls") pod "maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" (UID: "77751960-fc94-4513-a20c-c76a475834f0") : secret "maas-api-e2e-praxis-a-d61eae-serving-cert" not found Nov 03 00:38:42.592550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.592319 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg8wl\" (UniqueName: \"kubernetes.io/projected/77751960-fc94-4513-a20c-c76a475834f0-kube-api-access-vg8wl\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:42.592940 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.592920 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77751960-fc94-4513-a20c-c76a475834f0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:42.594748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.594731 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-metrics-certs\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:42.660364 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:42.660338 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg8wl\" (UniqueName: \"kubernetes.io/projected/77751960-fc94-4513-a20c-c76a475834f0-kube-api-access-vg8wl\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:43.097639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:43.097558 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:43.100140 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:43.100115 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:43.278422 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:43.278388 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:43.616091 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:43.616010 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx"] Nov 03 00:38:43.616493 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:43.616198 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" podUID="135bebde-c159-440a-8b6c-2ae327db3d7e" containerName="discovery" containerID="cri-o://8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50" gracePeriod=30 Nov 03 00:38:43.716665 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:38:43.716637 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77751960_fc94_4513_a20c_c76a475834f0.slice/crio-02ba1ac19c97278354b3b2eb5660a4d70823a5b53e9b07f7ac9106924a248972 WatchSource:0}: Error finding container 02ba1ac19c97278354b3b2eb5660a4d70823a5b53e9b07f7ac9106924a248972: Status 404 returned error can't find the container with id 02ba1ac19c97278354b3b2eb5660a4d70823a5b53e9b07f7ac9106924a248972 Nov 03 00:38:43.720577 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:43.720554 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx"] Nov 03 00:38:43.975718 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:43.975689 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:38:44.110053 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.110022 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-dns-cert\") pod \"135bebde-c159-440a-8b6c-2ae327db3d7e\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " Nov 03 00:38:44.110240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.110091 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-ca-configmap\") pod \"135bebde-c159-440a-8b6c-2ae327db3d7e\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " Nov 03 00:38:44.110314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.110284 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-cacerts\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-cacerts\") pod \"135bebde-c159-440a-8b6c-2ae327db3d7e\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " Nov 03 00:38:44.110401 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.110382 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135bebde-c159-440a-8b6c-2ae327db3d7e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/135bebde-c159-440a-8b6c-2ae327db3d7e-local-certs\") pod \"135bebde-c159-440a-8b6c-2ae327db3d7e\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " Nov 03 00:38:44.110495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.110473 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-ca-configmap" pod "135bebde-c159-440a-8b6c-2ae327db3d7e" (UID: "135bebde-c159-440a-8b6c-2ae327db3d7e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:38:44.110561 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.110476 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-kube-api-access-jjxjx\" (UniqueName: \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-kube-api-access-jjxjx\") pod \"135bebde-c159-440a-8b6c-2ae327db3d7e\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " Nov 03 00:38:44.110561 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.110548 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-kubeconfig\") pod \"135bebde-c159-440a-8b6c-2ae327db3d7e\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " Nov 03 00:38:44.110670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.110637 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-token\" (UniqueName: \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-token\") pod \"135bebde-c159-440a-8b6c-2ae327db3d7e\" (UID: \"135bebde-c159-440a-8b6c-2ae327db3d7e\") " Nov 03 00:38:44.111403 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.111333 4304 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-ca-configmap\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:44.113413 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.113291 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-dns-cert" pod "135bebde-c159-440a-8b6c-2ae327db3d7e" (UID: "135bebde-c159-440a-8b6c-2ae327db3d7e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:38:44.113413 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.113329 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135bebde-c159-440a-8b6c-2ae327db3d7e-local-certs" pod "135bebde-c159-440a-8b6c-2ae327db3d7e" (UID: "135bebde-c159-440a-8b6c-2ae327db3d7e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:38:44.113413 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.113335 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-kube-api-access-jjxjx" pod "135bebde-c159-440a-8b6c-2ae327db3d7e" (UID: "135bebde-c159-440a-8b6c-2ae327db3d7e"). InnerVolumeSpecName "kube-api-access-jjxjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:38:44.113641 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.113614 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-token" pod "135bebde-c159-440a-8b6c-2ae327db3d7e" (UID: "135bebde-c159-440a-8b6c-2ae327db3d7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:38:44.113893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.113863 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-kubeconfig" pod "135bebde-c159-440a-8b6c-2ae327db3d7e" (UID: "135bebde-c159-440a-8b6c-2ae327db3d7e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:38:44.113893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.113874 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-cacerts" pod "135bebde-c159-440a-8b6c-2ae327db3d7e" (UID: "135bebde-c159-440a-8b6c-2ae327db3d7e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:38:44.212445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.212367 4304 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-kubeconfig\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:44.212445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.212396 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:44.212445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.212406 4304 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-istio-csr-dns-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:44.212445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.212414 4304 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/135bebde-c159-440a-8b6c-2ae327db3d7e-cacerts\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:44.212445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.212422 4304 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/135bebde-c159-440a-8b6c-2ae327db3d7e-local-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:44.212445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.212430 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjxjx\" (UniqueName: \"kubernetes.io/projected/135bebde-c159-440a-8b6c-2ae327db3d7e-kube-api-access-jjxjx\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:44.525572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.525474 4304 generic.go:356] "Generic (PLEG): container finished" podID="135bebde-c159-440a-8b6c-2ae327db3d7e" containerID="8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50" exitCode=0 Nov 03 00:38:44.525572 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.525531 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" Nov 03 00:38:44.525816 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.525575 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" event={"ID":"135bebde-c159-440a-8b6c-2ae327db3d7e","Type":"ContainerDied","Data":"8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50"} Nov 03 00:38:44.525816 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.525667 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx" event={"ID":"135bebde-c159-440a-8b6c-2ae327db3d7e","Type":"ContainerDied","Data":"cf9a563383fb0a3cb2808d75a15bee525253bbe9e1b95641b6d5c344fb057a6a"} Nov 03 00:38:44.525816 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.525694 4304 scope.go:122] "RemoveContainer" containerID="8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50" Nov 03 00:38:44.531975 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.531947 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" event={"ID":"77751960-fc94-4513-a20c-c76a475834f0","Type":"ContainerStarted","Data":"02ba1ac19c97278354b3b2eb5660a4d70823a5b53e9b07f7ac9106924a248972"} Nov 03 00:38:44.536333 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.536304 4304 scope.go:122] "RemoveContainer" containerID="8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50" Nov 03 00:38:44.536664 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:38:44.536644 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50\": container with ID starting with 8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50 not found: ID does not exist" containerID="8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50" Nov 03 00:38:44.536718 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.536673 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50"} err="failed to get container status \"8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50\": rpc error: code = NotFound desc = could not find container \"8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50\": container with ID starting with 8ccf2e5bc60caef8484f7216be373d9477a3de8aa4428cb91e68ccab364efe50 not found: ID does not exist" Nov 03 00:38:44.694555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.694525 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx"] Nov 03 00:38:44.738231 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.738202 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26xpx"] Nov 03 00:38:44.829553 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:44.829483 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135bebde-c159-440a-8b6c-2ae327db3d7e" path="/var/lib/kubelet/pods/135bebde-c159-440a-8b6c-2ae327db3d7e/volumes" Nov 03 00:38:45.469350 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:45.469318 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5"] Nov 03 00:38:45.469563 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:45.469535 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerName="istio-proxy" containerID="cri-o://f0e826d3b299db82a6ba5e86303ed6d33e1c4c0d42e0f6c30a1458fb49b5b4dd" gracePeriod=30 Nov 03 00:38:45.545823 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:45.542693 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" event={"ID":"77751960-fc94-4513-a20c-c76a475834f0","Type":"ContainerStarted","Data":"632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95"} Nov 03 00:38:45.545823 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:45.542918 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:45.701036 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:45.700976 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" podStartSLOduration=2.7520335989999998 podStartE2EDuration="3.700962095s" totalImagesPullingTime="948.928496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:45.69735594 +0000 UTC m=+2008.980913616" watchObservedRunningTime="2026-11-03 00:38:45.700962095 +0000 UTC m=+2008.984519769" Nov 03 00:38:47.461946 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.461908 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h"] Nov 03 00:38:47.462901 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.462878 4304 cpu_manager.go:414] "containerMap: removing container" podUID="135bebde-c159-440a-8b6c-2ae327db3d7e" containerName="discovery" Nov 03 00:38:47.463022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.462902 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135bebde-c159-440a-8b6c-2ae327db3d7e" containerName="discovery" Nov 03 00:38:47.463022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.463007 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="135bebde-c159-440a-8b6c-2ae327db3d7e" containerName="discovery" Nov 03 00:38:47.467958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.467933 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.599080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.567664 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-envoy\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.599080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.567768 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-podinfo\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.599080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.567827 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-credential-socket\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.599080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.567858 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnfgd\" (UniqueName: \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-kube-api-access-mnfgd\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.599080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.567893 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-certs\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.599080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.567935 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-token\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.599080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.567967 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istiod-ca-cert\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.599080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.568008 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-socket\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.599080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.568034 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-data\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.651839 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.651805 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9ada73-openshift-default-dockercfg-lc2rl\"" Nov 03 00:38:47.655180 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.655060 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:38:47.655180 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.655140 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:38:47.664245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.664213 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h"] Nov 03 00:38:47.670605 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.670573 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-podinfo\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.671885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.671855 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-credential-socket\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.672018 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.671901 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnfgd\" (UniqueName: \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-kube-api-access-mnfgd\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.672018 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.671942 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-certs\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.672018 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.671987 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-token\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.672250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.672021 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istiod-ca-cert\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.672250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.672065 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-socket\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.672250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.672090 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-data\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.672430 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.672317 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-envoy\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.673596 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.673575 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-certs\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.674061 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.674033 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-socket\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.674207 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.674183 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-data\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.674598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.674568 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-credential-socket\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.675031 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.675002 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istiod-ca-cert\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.675182 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.675055 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-podinfo\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.705804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.687172 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-envoy\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.731196 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.731122 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnfgd\" (UniqueName: \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-kube-api-access-mnfgd\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.733693 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.733621 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-token\") pod \"e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.781967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.781943 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:47.929561 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.929528 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:38:47.929716 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:47.929606 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 00:38:48.207393 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.207363 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h"] Nov 03 00:38:48.207677 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:38:48.207656 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb726b1_3930_4f25_b566_c30c2db2d6f6.slice/crio-df9eb66cfae057561f75ecedc4bcfa66f748fe9b11158e23d2f2971849c960fd WatchSource:0}: Error finding container df9eb66cfae057561f75ecedc4bcfa66f748fe9b11158e23d2f2971849c960fd: Status 404 returned error can't find the container with id df9eb66cfae057561f75ecedc4bcfa66f748fe9b11158e23d2f2971849c960fd Nov 03 00:38:48.210435 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.210405 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:48.210490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.210476 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:48.210525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.210509 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:48.679381 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.679350 4304 generic.go:356] "Generic (PLEG): container finished" podID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerID="2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd" exitCode=0 Nov 03 00:38:48.679789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.679419 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" event={"ID":"077ba44b-d69e-42f4-8511-b730c8c4855a","Type":"ContainerDied","Data":"2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd"} Nov 03 00:38:48.679905 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.679888 4304 scope.go:122] "RemoveContainer" containerID="2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd" Nov 03 00:38:48.681325 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.681305 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" event={"ID":"fdb726b1-3930-4f25-b566-c30c2db2d6f6","Type":"ContainerStarted","Data":"623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d"} Nov 03 00:38:48.681424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.681351 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" event={"ID":"fdb726b1-3930-4f25-b566-c30c2db2d6f6","Type":"ContainerStarted","Data":"df9eb66cfae057561f75ecedc4bcfa66f748fe9b11158e23d2f2971849c960fd"} Nov 03 00:38:48.681706 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.681680 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:48.681805 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.681768 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:48.681847 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.681839 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:48.782316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.782284 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:48.916464 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:48.916405 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" podStartSLOduration=1.916385155 podStartE2EDuration="1.916385155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:48.913394589 +0000 UTC m=+2012.196952265" watchObservedRunningTime="2026-11-03 00:38:48.916385155 +0000 UTC m=+2012.199942830" Nov 03 00:38:49.689196 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:49.689162 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" event={"ID":"077ba44b-d69e-42f4-8511-b730c8c4855a","Type":"ContainerStarted","Data":"15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803"} Nov 03 00:38:49.788015 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:49.787986 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:50.313845 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.313811 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:38:50.314038 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.314016 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:38:50.314119 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.314076 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 00:38:50.700567 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.700532 4304 generic.go:356] "Generic (PLEG): container finished" podID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerID="f0e826d3b299db82a6ba5e86303ed6d33e1c4c0d42e0f6c30a1458fb49b5b4dd" exitCode=0 Nov 03 00:38:50.701289 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.701267 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" event={"ID":"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200","Type":"ContainerDied","Data":"f0e826d3b299db82a6ba5e86303ed6d33e1c4c0d42e0f6c30a1458fb49b5b4dd"} Nov 03 00:38:50.702383 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.701937 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:50.703680 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.703423 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:38:50.786371 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.786347 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:38:50.813245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813212 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istiod-ca-cert\") pod \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " Nov 03 00:38:50.813388 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813299 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-token\" (UniqueName: \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-token\") pod \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " Nov 03 00:38:50.813388 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813320 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-envoy\") pod \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " Nov 03 00:38:50.813473 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813386 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-socket\") pod \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " Nov 03 00:38:50.813473 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813460 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-podinfo\") pod \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " Nov 03 00:38:50.813556 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813496 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-certs\") pod \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " Nov 03 00:38:50.813609 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813551 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-data\") pod \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " Nov 03 00:38:50.813692 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813663 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-credential-socket\") pod \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " Nov 03 00:38:50.813750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813597 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istiod-ca-cert" pod "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" (UID: "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:38:50.813750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813717 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-kube-api-access-pvktn\" (UniqueName: \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-kube-api-access-pvktn\") pod \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\" (UID: \"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200\") " Nov 03 00:38:50.813750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813673 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-socket" pod "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" (UID: "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:38:50.813937 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813874 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-data" pod "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" (UID: "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:38:50.814008 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.813967 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-credential-socket" pod "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" (UID: "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:38:50.814668 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.814631 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-certs" pod "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" (UID: "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:38:50.814896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.814874 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:50.814987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.814905 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:50.814987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.814921 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:50.814987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.814935 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:50.814987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.814950 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:50.815994 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.815964 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-envoy" pod "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" (UID: "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:38:50.816085 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.816028 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-token" pod "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" (UID: "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:38:50.816136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.816114 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-podinfo" pod "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" (UID: "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:38:50.816318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.816303 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-kube-api-access-pvktn" pod "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" (UID: "ecd63ece-35a8-4ac8-8678-9c6ff2b2f200"). InnerVolumeSpecName "kube-api-access-pvktn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:38:50.917135 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.917098 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:50.917312 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.917139 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvktn\" (UniqueName: \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-kube-api-access-pvktn\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:50.917312 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.917158 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:50.917312 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:50.917172 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:51.313338 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:51.313294 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:38:51.313524 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:51.313355 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 00:38:51.558000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:51.557973 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:38:51.707268 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:51.707229 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" event={"ID":"ecd63ece-35a8-4ac8-8678-9c6ff2b2f200","Type":"ContainerDied","Data":"aa841950e472456721b4ab9d94ce0683d8739f30dc47738cea126205729cc0b8"} Nov 03 00:38:51.707268 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:51.707266 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5" Nov 03 00:38:51.707804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:51.707303 4304 scope.go:122] "RemoveContainer" containerID="f0e826d3b299db82a6ba5e86303ed6d33e1c4c0d42e0f6c30a1458fb49b5b4dd" Nov 03 00:38:51.871472 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:51.871435 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5"] Nov 03 00:38:51.879870 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:51.879838 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-tq6d5"] Nov 03 00:38:52.314110 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:52.314078 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:38:52.314280 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:52.314133 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 00:38:52.850000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:52.849971 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" path="/var/lib/kubelet/pods/ecd63ece-35a8-4ac8-8678-9c6ff2b2f200/volumes" Nov 03 00:38:53.313725 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:53.313692 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:38:53.313931 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:53.313742 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 00:38:54.313293 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:54.313256 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:38:54.313732 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:54.313307 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 00:38:55.317437 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.317408 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:38:55.317831 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.317755 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:38:55.318393 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.318378 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:38:55.758136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.758107 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz"] Nov 03 00:38:55.758648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.758633 4304 cpu_manager.go:414] "containerMap: removing container" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerName="istio-proxy" Nov 03 00:38:55.758724 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.758649 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerName="istio-proxy" Nov 03 00:38:55.758791 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.758749 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd63ece-35a8-4ac8-8678-9c6ff2b2f200" containerName="istio-proxy" Nov 03 00:38:55.763655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.763635 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:38:55.816139 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.816090 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9ada73\"" Nov 03 00:38:55.816336 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.816239 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9ada73-dockercfg-r9hkc\"" Nov 03 00:38:55.819079 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.819060 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz"] Nov 03 00:38:55.899485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.899446 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1e9db6-48e7-4868-99d4-201349b1e2b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz\" (UID: \"9a1e9db6-48e7-4868-99d4-201349b1e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:38:55.899834 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.899813 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c874\" (UniqueName: \"kubernetes.io/projected/9a1e9db6-48e7-4868-99d4-201349b1e2b9-kube-api-access-2c874\") pod \"payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz\" (UID: \"9a1e9db6-48e7-4868-99d4-201349b1e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:38:55.924564 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.924521 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv"] Nov 03 00:38:55.929653 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.929613 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:38:55.982706 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:55.982673 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv"] Nov 03 00:38:56.001215 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.001180 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1e9db6-48e7-4868-99d4-201349b1e2b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz\" (UID: \"9a1e9db6-48e7-4868-99d4-201349b1e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:38:56.001370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.001314 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c874\" (UniqueName: \"kubernetes.io/projected/9a1e9db6-48e7-4868-99d4-201349b1e2b9-kube-api-access-2c874\") pod \"payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz\" (UID: \"9a1e9db6-48e7-4868-99d4-201349b1e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:38:56.002205 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.002172 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1e9db6-48e7-4868-99d4-201349b1e2b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz\" (UID: \"9a1e9db6-48e7-4868-99d4-201349b1e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:38:56.057811 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.057727 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c874\" (UniqueName: \"kubernetes.io/projected/9a1e9db6-48e7-4868-99d4-201349b1e2b9-kube-api-access-2c874\") pod \"payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz\" (UID: \"9a1e9db6-48e7-4868-99d4-201349b1e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:38:56.076477 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.076445 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:38:56.102389 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.102363 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade39df8-8a42-40ef-9652-1411ecdcb65f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv\" (UID: \"ade39df8-8a42-40ef-9652-1411ecdcb65f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:38:56.102512 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.102496 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tvd8\" (UniqueName: \"kubernetes.io/projected/ade39df8-8a42-40ef-9652-1411ecdcb65f-kube-api-access-4tvd8\") pod \"payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv\" (UID: \"ade39df8-8a42-40ef-9652-1411ecdcb65f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:38:56.203933 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.203896 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tvd8\" (UniqueName: \"kubernetes.io/projected/ade39df8-8a42-40ef-9652-1411ecdcb65f-kube-api-access-4tvd8\") pod \"payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv\" (UID: \"ade39df8-8a42-40ef-9652-1411ecdcb65f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:38:56.204126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.204064 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade39df8-8a42-40ef-9652-1411ecdcb65f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv\" (UID: \"ade39df8-8a42-40ef-9652-1411ecdcb65f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:38:56.205036 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.205010 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade39df8-8a42-40ef-9652-1411ecdcb65f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv\" (UID: \"ade39df8-8a42-40ef-9652-1411ecdcb65f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:38:56.260131 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.260100 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tvd8\" (UniqueName: \"kubernetes.io/projected/ade39df8-8a42-40ef-9652-1411ecdcb65f-kube-api-access-4tvd8\") pod \"payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv\" (UID: \"ade39df8-8a42-40ef-9652-1411ecdcb65f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:38:56.264006 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.263980 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:38:56.455316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.455292 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz"] Nov 03 00:38:56.457114 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:38:56.457082 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1e9db6_48e7_4868_99d4_201349b1e2b9.slice/crio-7a030b6ae9ec7e53144fc5f930f35be00c2637d3bf1d4010f04e653786716f2c WatchSource:0}: Error finding container 7a030b6ae9ec7e53144fc5f930f35be00c2637d3bf1d4010f04e653786716f2c: Status 404 returned error can't find the container with id 7a030b6ae9ec7e53144fc5f930f35be00c2637d3bf1d4010f04e653786716f2c Nov 03 00:38:56.635636 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.635606 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv"] Nov 03 00:38:56.637046 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:38:56.637015 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade39df8_8a42_40ef_9652_1411ecdcb65f.slice/crio-5524d441537a5f70320472b90f26b13a6bf689744451e5586261a04fb449e42d WatchSource:0}: Error finding container 5524d441537a5f70320472b90f26b13a6bf689744451e5586261a04fb449e42d: Status 404 returned error can't find the container with id 5524d441537a5f70320472b90f26b13a6bf689744451e5586261a04fb449e42d Nov 03 00:38:56.742687 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.742629 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" event={"ID":"9a1e9db6-48e7-4868-99d4-201349b1e2b9","Type":"ContainerStarted","Data":"fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed"} Nov 03 00:38:56.742888 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.742706 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" event={"ID":"9a1e9db6-48e7-4868-99d4-201349b1e2b9","Type":"ContainerStarted","Data":"7a030b6ae9ec7e53144fc5f930f35be00c2637d3bf1d4010f04e653786716f2c"} Nov 03 00:38:56.742888 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.742735 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:38:56.744301 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.744277 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" event={"ID":"ade39df8-8a42-40ef-9652-1411ecdcb65f","Type":"ContainerStarted","Data":"6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d"} Nov 03 00:38:56.744436 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.744326 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" event={"ID":"ade39df8-8a42-40ef-9652-1411ecdcb65f","Type":"ContainerStarted","Data":"5524d441537a5f70320472b90f26b13a6bf689744451e5586261a04fb449e42d"} Nov 03 00:38:56.744496 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.744453 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:38:56.855432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.854972 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" podStartSLOduration=1.854954814 podStartE2EDuration="1.854954814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:56.854363712 +0000 UTC m=+2020.137921396" watchObservedRunningTime="2026-11-03 00:38:56.854954814 +0000 UTC m=+2020.138512490" Nov 03 00:38:56.970872 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:38:56.970804 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" podStartSLOduration=1.970768769 podStartE2EDuration="1.970768769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:56.967968582 +0000 UTC m=+2020.251526259" watchObservedRunningTime="2026-11-03 00:38:56.970768769 +0000 UTC m=+2020.254326444" Nov 03 00:38:58.655651 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:38:58.655617 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:39:07.753334 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:07.753306 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:39:07.753754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:07.753372 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:39:12.720623 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:12.720588 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk"] Nov 03 00:39:12.721126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:12.720852 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" containerID="cri-o://77c13092cebbc23c4383bdfba80568f23dbf7847a04454d80d2efd1cd00bad0f" gracePeriod=30 Nov 03 00:39:14.381547 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.381515 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm"] Nov 03 00:39:14.385589 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.385572 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:39:14.423796 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.423746 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnt7s\" (UniqueName: \"kubernetes.io/projected/01f9ade9-8108-4870-a6ae-47007de2163f-kube-api-access-jnt7s\") pod \"payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm\" (UID: \"01f9ade9-8108-4870-a6ae-47007de2163f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:39:14.424000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.423975 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f9ade9-8108-4870-a6ae-47007de2163f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm\" (UID: \"01f9ade9-8108-4870-a6ae-47007de2163f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:39:14.439150 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.439088 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-011885\"" Nov 03 00:39:14.488637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.488606 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-011885-dockercfg-wt4wd\"" Nov 03 00:39:14.491943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.491919 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm"] Nov 03 00:39:14.525402 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.525374 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f9ade9-8108-4870-a6ae-47007de2163f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm\" (UID: \"01f9ade9-8108-4870-a6ae-47007de2163f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:39:14.525683 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.525435 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnt7s\" (UniqueName: \"kubernetes.io/projected/01f9ade9-8108-4870-a6ae-47007de2163f-kube-api-access-jnt7s\") pod \"payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm\" (UID: \"01f9ade9-8108-4870-a6ae-47007de2163f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:39:14.526062 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.526043 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f9ade9-8108-4870-a6ae-47007de2163f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm\" (UID: \"01f9ade9-8108-4870-a6ae-47007de2163f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:39:14.581748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.581727 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnt7s\" (UniqueName: \"kubernetes.io/projected/01f9ade9-8108-4870-a6ae-47007de2163f-kube-api-access-jnt7s\") pod \"payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm\" (UID: \"01f9ade9-8108-4870-a6ae-47007de2163f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:39:14.704497 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:14.704470 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:39:15.078975 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.078944 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm"] Nov 03 00:39:15.079729 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:39:15.079708 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f9ade9_8108_4870_a6ae_47007de2163f.slice/crio-dd2deeaf1f5d314256bfe0047eb6d1e46e2552533fc3364f1a3ae25c059eadd4 WatchSource:0}: Error finding container dd2deeaf1f5d314256bfe0047eb6d1e46e2552533fc3364f1a3ae25c059eadd4: Status 404 returned error can't find the container with id dd2deeaf1f5d314256bfe0047eb6d1e46e2552533fc3364f1a3ae25c059eadd4 Nov 03 00:39:15.156030 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.156006 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64"] Nov 03 00:39:15.159888 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.159873 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:39:15.216316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.216274 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64"] Nov 03 00:39:15.232656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.232611 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4902af84-c86b-46c7-a4f3-e0bac046ca77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-011885-644fff48b-n7z64\" (UID: \"4902af84-c86b-46c7-a4f3-e0bac046ca77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:39:15.232829 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.232724 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngfgs\" (UniqueName: \"kubernetes.io/projected/4902af84-c86b-46c7-a4f3-e0bac046ca77-kube-api-access-ngfgs\") pod \"payload-processing-e2e-ipp-b-011885-644fff48b-n7z64\" (UID: \"4902af84-c86b-46c7-a4f3-e0bac046ca77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:39:15.333446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.333375 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4902af84-c86b-46c7-a4f3-e0bac046ca77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-011885-644fff48b-n7z64\" (UID: \"4902af84-c86b-46c7-a4f3-e0bac046ca77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:39:15.333446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.333426 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfgs\" (UniqueName: \"kubernetes.io/projected/4902af84-c86b-46c7-a4f3-e0bac046ca77-kube-api-access-ngfgs\") pod \"payload-processing-e2e-ipp-b-011885-644fff48b-n7z64\" (UID: \"4902af84-c86b-46c7-a4f3-e0bac046ca77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:39:15.334116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.334095 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4902af84-c86b-46c7-a4f3-e0bac046ca77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-011885-644fff48b-n7z64\" (UID: \"4902af84-c86b-46c7-a4f3-e0bac046ca77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:39:15.390021 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.389989 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngfgs\" (UniqueName: \"kubernetes.io/projected/4902af84-c86b-46c7-a4f3-e0bac046ca77-kube-api-access-ngfgs\") pod \"payload-processing-e2e-ipp-b-011885-644fff48b-n7z64\" (UID: \"4902af84-c86b-46c7-a4f3-e0bac046ca77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:39:15.473672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.473634 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:39:15.849555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.849531 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64"] Nov 03 00:39:15.850869 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:39:15.850844 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4902af84_c86b_46c7_a4f3_e0bac046ca77.slice/crio-298feabffa76bf68e27fc3c70a03ad8e1e867c347bfe6166ca68562ef80956b0 WatchSource:0}: Error finding container 298feabffa76bf68e27fc3c70a03ad8e1e867c347bfe6166ca68562ef80956b0: Status 404 returned error can't find the container with id 298feabffa76bf68e27fc3c70a03ad8e1e867c347bfe6166ca68562ef80956b0 Nov 03 00:39:15.875245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.875218 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" event={"ID":"4902af84-c86b-46c7-a4f3-e0bac046ca77","Type":"ContainerStarted","Data":"298feabffa76bf68e27fc3c70a03ad8e1e867c347bfe6166ca68562ef80956b0"} Nov 03 00:39:15.876904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.876874 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" event={"ID":"01f9ade9-8108-4870-a6ae-47007de2163f","Type":"ContainerStarted","Data":"d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366"} Nov 03 00:39:15.877022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.876933 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" event={"ID":"01f9ade9-8108-4870-a6ae-47007de2163f","Type":"ContainerStarted","Data":"dd2deeaf1f5d314256bfe0047eb6d1e46e2552533fc3364f1a3ae25c059eadd4"} Nov 03 00:39:15.877179 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.877135 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:39:15.992726 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:15.992672 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" podStartSLOduration=1.992657495 podStartE2EDuration="1.992657495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:39:15.989911364 +0000 UTC m=+2039.273469039" watchObservedRunningTime="2026-11-03 00:39:15.992657495 +0000 UTC m=+2039.276215168" Nov 03 00:39:16.057237 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:16.057200 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:39:16.057418 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:16.057264 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:39:16.883655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:16.883620 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" event={"ID":"4902af84-c86b-46c7-a4f3-e0bac046ca77","Type":"ContainerStarted","Data":"f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d"} Nov 03 00:39:16.884081 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:16.883685 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:39:16.998002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:16.997948 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" podStartSLOduration=1.997929681 podStartE2EDuration="1.997929681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:39:16.996288352 +0000 UTC m=+2040.279846020" watchObservedRunningTime="2026-11-03 00:39:16.997929681 +0000 UTC m=+2040.281487360" Nov 03 00:39:17.895444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:17.895417 4304 generic.go:356] "Generic (PLEG): container finished" podID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerID="77c13092cebbc23c4383bdfba80568f23dbf7847a04454d80d2efd1cd00bad0f" exitCode=0 Nov 03 00:39:17.895811 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:17.895488 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" event={"ID":"ea622cbd-cfbc-44d5-81be-d412de79058e","Type":"ContainerDied","Data":"77c13092cebbc23c4383bdfba80568f23dbf7847a04454d80d2efd1cd00bad0f"} Nov 03 00:39:17.980277 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:17.980252 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:39:18.069373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069299 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea622cbd-cfbc-44d5-81be-d412de79058e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea622cbd-cfbc-44d5-81be-d412de79058e-istiod-ca-cert\") pod \"ea622cbd-cfbc-44d5-81be-d412de79058e\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " Nov 03 00:39:18.069373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069366 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-envoy\") pod \"ea622cbd-cfbc-44d5-81be-d412de79058e\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " Nov 03 00:39:18.069592 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069391 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-socket\") pod \"ea622cbd-cfbc-44d5-81be-d412de79058e\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " Nov 03 00:39:18.069592 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069436 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-token\" (UniqueName: \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-token\") pod \"ea622cbd-cfbc-44d5-81be-d412de79058e\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " Nov 03 00:39:18.069592 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069502 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-certs\") pod \"ea622cbd-cfbc-44d5-81be-d412de79058e\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " Nov 03 00:39:18.069592 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069546 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-podinfo\") pod \"ea622cbd-cfbc-44d5-81be-d412de79058e\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " Nov 03 00:39:18.069592 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069577 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-kube-api-access-4vh2d\" (UniqueName: \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-kube-api-access-4vh2d\") pod \"ea622cbd-cfbc-44d5-81be-d412de79058e\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " Nov 03 00:39:18.069889 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069647 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-credential-socket\") pod \"ea622cbd-cfbc-44d5-81be-d412de79058e\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " Nov 03 00:39:18.069889 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069686 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-socket" pod "ea622cbd-cfbc-44d5-81be-d412de79058e" (UID: "ea622cbd-cfbc-44d5-81be-d412de79058e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:18.069889 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069712 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea622cbd-cfbc-44d5-81be-d412de79058e-istiod-ca-cert" pod "ea622cbd-cfbc-44d5-81be-d412de79058e" (UID: "ea622cbd-cfbc-44d5-81be-d412de79058e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:39:18.069889 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069720 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-data\") pod \"ea622cbd-cfbc-44d5-81be-d412de79058e\" (UID: \"ea622cbd-cfbc-44d5-81be-d412de79058e\") " Nov 03 00:39:18.069889 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069841 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-certs" pod "ea622cbd-cfbc-44d5-81be-d412de79058e" (UID: "ea622cbd-cfbc-44d5-81be-d412de79058e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:18.070119 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069939 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-data" pod "ea622cbd-cfbc-44d5-81be-d412de79058e" (UID: "ea622cbd-cfbc-44d5-81be-d412de79058e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:18.070119 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.069989 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-credential-socket" pod "ea622cbd-cfbc-44d5-81be-d412de79058e" (UID: "ea622cbd-cfbc-44d5-81be-d412de79058e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:18.070523 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.070502 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:18.070576 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.070532 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea622cbd-cfbc-44d5-81be-d412de79058e-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:18.070576 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.070549 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:18.070576 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.070563 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:18.070668 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.070577 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:18.071758 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.071737 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-token" pod "ea622cbd-cfbc-44d5-81be-d412de79058e" (UID: "ea622cbd-cfbc-44d5-81be-d412de79058e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:18.071898 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.071876 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-kube-api-access-4vh2d" pod "ea622cbd-cfbc-44d5-81be-d412de79058e" (UID: "ea622cbd-cfbc-44d5-81be-d412de79058e"). InnerVolumeSpecName "kube-api-access-4vh2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:18.072196 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.072175 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-envoy" pod "ea622cbd-cfbc-44d5-81be-d412de79058e" (UID: "ea622cbd-cfbc-44d5-81be-d412de79058e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:18.072285 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.072222 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-podinfo" pod "ea622cbd-cfbc-44d5-81be-d412de79058e" (UID: "ea622cbd-cfbc-44d5-81be-d412de79058e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:39:18.172233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.172202 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:18.172233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.172231 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:18.172233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.172241 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea622cbd-cfbc-44d5-81be-d412de79058e-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:18.172465 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.172251 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vh2d\" (UniqueName: \"kubernetes.io/projected/ea622cbd-cfbc-44d5-81be-d412de79058e-kube-api-access-4vh2d\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:18.903262 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.903230 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" event={"ID":"ea622cbd-cfbc-44d5-81be-d412de79058e","Type":"ContainerDied","Data":"2400b3c462fb49897e67985df11be87bb8043cd3a2101a603e7809d423d10438"} Nov 03 00:39:18.903262 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.903258 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk" Nov 03 00:39:18.903710 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:18.903290 4304 scope.go:122] "RemoveContainer" containerID="77c13092cebbc23c4383bdfba80568f23dbf7847a04454d80d2efd1cd00bad0f" Nov 03 00:39:19.116715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:19.116644 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk"] Nov 03 00:39:19.123068 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:19.123041 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-bgbdk"] Nov 03 00:39:20.451462 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.451428 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2"] Nov 03 00:39:20.451993 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.451979 4304 cpu_manager.go:414] "containerMap: removing container" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" Nov 03 00:39:20.452043 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.451993 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" Nov 03 00:39:20.452087 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.452078 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" containerName="istio-proxy" Nov 03 00:39:20.456875 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.456855 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.559023 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.558994 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ff10d6-serving-cert\"" Nov 03 00:39:20.562347 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.562323 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2"] Nov 03 00:39:20.602116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.602086 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-metrics-certs\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.602240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.602124 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef67aae-a041-4563-866c-e803dde848fd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.602295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.602275 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7fcj\" (UniqueName: \"kubernetes.io/projected/bef67aae-a041-4563-866c-e803dde848fd-kube-api-access-b7fcj\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.602413 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.602399 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.703192 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.703101 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7fcj\" (UniqueName: \"kubernetes.io/projected/bef67aae-a041-4563-866c-e803dde848fd-kube-api-access-b7fcj\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.703356 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.703246 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.703356 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.703301 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-metrics-certs\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.703356 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.703335 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef67aae-a041-4563-866c-e803dde848fd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.703533 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:39:20.703460 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-serving-cert: secret "maas-api-e2e-praxis-a-ff10d6-serving-cert" not found Nov 03 00:39:20.703583 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:39:20.703531 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-maas-api-tls podName:bef67aae-a041-4563-866c-e803dde848fd nodeName:}" failed. No retries permitted until 2026-11-03 00:39:21.203510449 +0000 UTC m=+2044.487068127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-maas-api-tls") pod "maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" (UID: "bef67aae-a041-4563-866c-e803dde848fd") : secret "maas-api-e2e-praxis-a-ff10d6-serving-cert" not found Nov 03 00:39:20.704098 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.704070 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef67aae-a041-4563-866c-e803dde848fd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.705921 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.705902 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-metrics-certs\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.761209 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.761179 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7fcj\" (UniqueName: \"kubernetes.io/projected/bef67aae-a041-4563-866c-e803dde848fd-kube-api-access-b7fcj\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:20.829678 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:20.829635 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea622cbd-cfbc-44d5-81be-d412de79058e" path="/var/lib/kubelet/pods/ea622cbd-cfbc-44d5-81be-d412de79058e/volumes" Nov 03 00:39:21.209105 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:21.209068 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:21.211450 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:21.211426 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:21.370282 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:21.370238 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:21.779325 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:21.779292 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2"] Nov 03 00:39:21.780190 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:39:21.780161 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef67aae_a041_4563_866c_e803dde848fd.slice/crio-ad779c04e263b2c1d703a3b95b0e3e5f8b1c097a7dd866f1e5fbe14906698bb3 WatchSource:0}: Error finding container ad779c04e263b2c1d703a3b95b0e3e5f8b1c097a7dd866f1e5fbe14906698bb3: Status 404 returned error can't find the container with id ad779c04e263b2c1d703a3b95b0e3e5f8b1c097a7dd866f1e5fbe14906698bb3 Nov 03 00:39:21.923398 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:21.923367 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" event={"ID":"bef67aae-a041-4563-866c-e803dde848fd","Type":"ContainerStarted","Data":"ad779c04e263b2c1d703a3b95b0e3e5f8b1c097a7dd866f1e5fbe14906698bb3"} Nov 03 00:39:22.940366 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:22.940328 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" event={"ID":"bef67aae-a041-4563-866c-e803dde848fd","Type":"ContainerStarted","Data":"92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777"} Nov 03 00:39:22.940842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:22.940426 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:23.054380 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:23.054326 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" podStartSLOduration=2.205276943 podStartE2EDuration="3.054309643s" totalImagesPullingTime="849.0327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:39:23.053693142 +0000 UTC m=+2046.337250815" watchObservedRunningTime="2026-11-03 00:39:23.054309643 +0000 UTC m=+2046.337867318" Nov 03 00:39:26.885513 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:26.885486 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:39:27.898381 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:27.898353 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:39:28.654862 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:39:28.654828 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:39:28.951274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:28.951206 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:39:58.655606 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:39:58.655571 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:39:59.238616 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.238578 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87"] Nov 03 00:39:59.243533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.243513 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.296193 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.296158 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9ada73-serving-cert\"" Nov 03 00:39:59.348971 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.348936 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87"] Nov 03 00:39:59.355645 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.355622 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-metrics-certs\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.355768 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.355685 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5212a935-d625-48fd-a814-171f2cf68758-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.355904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.355882 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.355968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.355932 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxrcl\" (UniqueName: \"kubernetes.io/projected/5212a935-d625-48fd-a814-171f2cf68758-kube-api-access-wxrcl\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.457151 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.457123 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.457347 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.457163 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxrcl\" (UniqueName: \"kubernetes.io/projected/5212a935-d625-48fd-a814-171f2cf68758-kube-api-access-wxrcl\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.457347 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.457250 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-metrics-certs\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.457347 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:39:59.457276 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-serving-cert: secret "maas-api-e2e-praxis-b-9ada73-serving-cert" not found Nov 03 00:39:59.457347 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.457310 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5212a935-d625-48fd-a814-171f2cf68758-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.457580 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:39:59.457356 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-maas-api-tls podName:5212a935-d625-48fd-a814-171f2cf68758 nodeName:}" failed. No retries permitted until 2026-11-03 00:39:59.957335733 +0000 UTC m=+2083.240893402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-maas-api-tls") pod "maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" (UID: "5212a935-d625-48fd-a814-171f2cf68758") : secret "maas-api-e2e-praxis-b-9ada73-serving-cert" not found Nov 03 00:39:59.457987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.457967 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5212a935-d625-48fd-a814-171f2cf68758-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.459900 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.459881 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-metrics-certs\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.514251 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.514182 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxrcl\" (UniqueName: \"kubernetes.io/projected/5212a935-d625-48fd-a814-171f2cf68758-kube-api-access-wxrcl\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.964142 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.964102 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:39:59.966899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:39:59.966870 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:40:00.156981 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:00.156948 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:40:00.564479 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:00.564452 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87"] Nov 03 00:40:00.565580 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:40:00.565555 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5212a935_d625_48fd_a814_171f2cf68758.slice/crio-5a8956a97acc3b2f8cd0c0e46678c29c29cd1f3e2e5119f6da01e7ebc2bb8bf2 WatchSource:0}: Error finding container 5a8956a97acc3b2f8cd0c0e46678c29c29cd1f3e2e5119f6da01e7ebc2bb8bf2: Status 404 returned error can't find the container with id 5a8956a97acc3b2f8cd0c0e46678c29c29cd1f3e2e5119f6da01e7ebc2bb8bf2 Nov 03 00:40:01.187383 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:01.187348 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" event={"ID":"5212a935-d625-48fd-a814-171f2cf68758","Type":"ContainerStarted","Data":"5a8956a97acc3b2f8cd0c0e46678c29c29cd1f3e2e5119f6da01e7ebc2bb8bf2"} Nov 03 00:40:02.195016 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:02.194961 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" event={"ID":"5212a935-d625-48fd-a814-171f2cf68758","Type":"ContainerStarted","Data":"3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9"} Nov 03 00:40:02.195503 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:02.195061 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:40:02.311628 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:02.311566 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" podStartSLOduration=2.297019821 podStartE2EDuration="3.311551947s" totalImagesPullingTime="1.014532126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:02.308041248 +0000 UTC m=+2085.591598927" watchObservedRunningTime="2026-11-03 00:40:02.311551947 +0000 UTC m=+2085.595109621" Nov 03 00:40:08.207343 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:08.207315 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:40:08.977808 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:08.976606 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m"] Nov 03 00:40:08.983984 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:08.983952 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:40:09.034937 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:09.034905 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m"] Nov 03 00:40:09.036872 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:09.036845 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bdd194\"" Nov 03 00:40:09.121562 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:09.121534 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bdd194-dockercfg-sp6j5\"" Nov 03 00:40:09.146793 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:09.146740 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr6k2\" (UniqueName: \"kubernetes.io/projected/c2331374-06e2-49fe-8f5f-5501302327ab-kube-api-access-dr6k2\") pod \"payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m\" (UID: \"c2331374-06e2-49fe-8f5f-5501302327ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:40:09.146982 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:09.146960 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2331374-06e2-49fe-8f5f-5501302327ab-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m\" (UID: \"c2331374-06e2-49fe-8f5f-5501302327ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:40:09.248624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:09.248545 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6k2\" (UniqueName: \"kubernetes.io/projected/c2331374-06e2-49fe-8f5f-5501302327ab-kube-api-access-dr6k2\") pod \"payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m\" (UID: \"c2331374-06e2-49fe-8f5f-5501302327ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:40:09.249045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:09.248634 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2331374-06e2-49fe-8f5f-5501302327ab-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m\" (UID: \"c2331374-06e2-49fe-8f5f-5501302327ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:40:09.249341 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:09.249322 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2331374-06e2-49fe-8f5f-5501302327ab-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m\" (UID: \"c2331374-06e2-49fe-8f5f-5501302327ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:40:09.304943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:09.304912 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6k2\" (UniqueName: \"kubernetes.io/projected/c2331374-06e2-49fe-8f5f-5501302327ab-kube-api-access-dr6k2\") pod \"payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m\" (UID: \"c2331374-06e2-49fe-8f5f-5501302327ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:40:09.306238 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:09.306216 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:40:09.685222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:09.685193 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m"] Nov 03 00:40:09.686403 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:40:09.686380 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2331374_06e2_49fe_8f5f_5501302327ab.slice/crio-d025ed0a4dd476933e64538b2c2be9bc81b42e5b8893e5c9ddcf82bb7bf6ddb3 WatchSource:0}: Error finding container d025ed0a4dd476933e64538b2c2be9bc81b42e5b8893e5c9ddcf82bb7bf6ddb3: Status 404 returned error can't find the container with id d025ed0a4dd476933e64538b2c2be9bc81b42e5b8893e5c9ddcf82bb7bf6ddb3 Nov 03 00:40:10.247310 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:10.247269 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" event={"ID":"c2331374-06e2-49fe-8f5f-5501302327ab","Type":"ContainerStarted","Data":"71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e"} Nov 03 00:40:10.247487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:10.247334 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" event={"ID":"c2331374-06e2-49fe-8f5f-5501302327ab","Type":"ContainerStarted","Data":"d025ed0a4dd476933e64538b2c2be9bc81b42e5b8893e5c9ddcf82bb7bf6ddb3"} Nov 03 00:40:10.247487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:10.247357 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:40:10.361875 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:10.361816 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" podStartSLOduration=2.361802311 podStartE2EDuration="2.361802311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:10.359250016 +0000 UTC m=+2093.642807691" watchObservedRunningTime="2026-11-03 00:40:10.361802311 +0000 UTC m=+2093.645359985" Nov 03 00:40:20.585104 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:20.585066 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-011885-644fff48b-n7z64_4902af84-c86b-46c7-a4f3-e0bac046ca77/payload-processing/0.log" Nov 03 00:40:20.822803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:20.818618 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-011885-644fff48b-n7z64_4902af84-c86b-46c7-a4f3-e0bac046ca77/payload-processing/0.log" Nov 03 00:40:21.258626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:21.258603 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:40:28.656126 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:40:28.656089 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:40:47.516682 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:47.516605 4304 generic.go:356] "Generic (PLEG): container finished" podID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerID="623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d" exitCode=0 Nov 03 00:40:47.516682 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:47.516656 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" event={"ID":"fdb726b1-3930-4f25-b566-c30c2db2d6f6","Type":"ContainerDied","Data":"623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d"} Nov 03 00:40:47.517210 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:47.517080 4304 scope.go:122] "RemoveContainer" containerID="623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d" Nov 03 00:40:47.518626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:47.518590 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:40:47.518734 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:47.518681 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:40:47.518825 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:47.518732 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:40:48.543964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:48.543927 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" event={"ID":"fdb726b1-3930-4f25-b566-c30c2db2d6f6","Type":"ContainerStarted","Data":"67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd"} Nov 03 00:40:48.783095 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:48.783060 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:40:48.784197 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:48.784174 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:40:48.784313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:48.784223 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 00:40:49.783000 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:49.782963 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:40:49.783399 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:49.783026 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 00:40:50.702236 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:50.702196 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:40:50.783048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:50.783012 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:40:50.783429 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:50.783067 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 00:40:51.783187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:51.783150 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:40:51.783591 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:51.783216 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 00:40:52.782495 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:52.782457 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:40:52.782682 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:52.782513 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 00:40:53.782743 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:53.782708 4304 patch_prober.go:29] interesting pod/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:40:53.783194 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:53.782802 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 00:40:54.786929 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:54.786899 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:40:54.787985 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:40:54.787967 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:40:58.654763 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:40:58.654728 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:41:28.654718 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:41:28.654679 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:41:43.068432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:43.068398 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7"] Nov 03 00:41:43.068882 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:43.068636 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" containerID="cri-o://15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803" gracePeriod=30 Nov 03 00:41:48.337232 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.337174 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:41:48.343468 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.343445 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-kube-api-access-fzvbg\" (UniqueName: \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-kube-api-access-fzvbg\") pod \"077ba44b-d69e-42f4-8511-b730c8c4855a\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " Nov 03 00:41:48.343608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.343483 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-podinfo\") pod \"077ba44b-d69e-42f4-8511-b730c8c4855a\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " Nov 03 00:41:48.343608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.343519 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-data\") pod \"077ba44b-d69e-42f4-8511-b730c8c4855a\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " Nov 03 00:41:48.343608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.343538 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-socket\") pod \"077ba44b-d69e-42f4-8511-b730c8c4855a\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " Nov 03 00:41:48.343608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.343564 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-envoy\") pod \"077ba44b-d69e-42f4-8511-b730c8c4855a\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " Nov 03 00:41:48.343608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.343600 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/077ba44b-d69e-42f4-8511-b730c8c4855a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077ba44b-d69e-42f4-8511-b730c8c4855a-istiod-ca-cert\") pod \"077ba44b-d69e-42f4-8511-b730c8c4855a\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " Nov 03 00:41:48.343909 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.343681 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-certs\") pod \"077ba44b-d69e-42f4-8511-b730c8c4855a\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " Nov 03 00:41:48.343909 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.343824 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-credential-socket\") pod \"077ba44b-d69e-42f4-8511-b730c8c4855a\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " Nov 03 00:41:48.344027 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.343912 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-token\" (UniqueName: \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-token\") pod \"077ba44b-d69e-42f4-8511-b730c8c4855a\" (UID: \"077ba44b-d69e-42f4-8511-b730c8c4855a\") " Nov 03 00:41:48.344027 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.343913 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-data" pod "077ba44b-d69e-42f4-8511-b730c8c4855a" (UID: "077ba44b-d69e-42f4-8511-b730c8c4855a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:48.344027 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.343944 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-socket" pod "077ba44b-d69e-42f4-8511-b730c8c4855a" (UID: "077ba44b-d69e-42f4-8511-b730c8c4855a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:48.344414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.344156 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-certs" pod "077ba44b-d69e-42f4-8511-b730c8c4855a" (UID: "077ba44b-d69e-42f4-8511-b730c8c4855a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:48.344414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.344177 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077ba44b-d69e-42f4-8511-b730c8c4855a-istiod-ca-cert" pod "077ba44b-d69e-42f4-8511-b730c8c4855a" (UID: "077ba44b-d69e-42f4-8511-b730c8c4855a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:48.344414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.344171 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-credential-socket" pod "077ba44b-d69e-42f4-8511-b730c8c4855a" (UID: "077ba44b-d69e-42f4-8511-b730c8c4855a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:48.344595 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.344573 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:48.344665 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.344603 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:48.344665 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.344619 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077ba44b-d69e-42f4-8511-b730c8c4855a-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:48.344665 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.344633 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:48.344665 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.344648 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:48.345946 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.345921 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-envoy" pod "077ba44b-d69e-42f4-8511-b730c8c4855a" (UID: "077ba44b-d69e-42f4-8511-b730c8c4855a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:48.346067 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.345994 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-kube-api-access-fzvbg" pod "077ba44b-d69e-42f4-8511-b730c8c4855a" (UID: "077ba44b-d69e-42f4-8511-b730c8c4855a"). InnerVolumeSpecName "kube-api-access-fzvbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:48.346067 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.346045 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-podinfo" pod "077ba44b-d69e-42f4-8511-b730c8c4855a" (UID: "077ba44b-d69e-42f4-8511-b730c8c4855a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:41:48.346274 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.346259 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-token" pod "077ba44b-d69e-42f4-8511-b730c8c4855a" (UID: "077ba44b-d69e-42f4-8511-b730c8c4855a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:48.446225 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.446178 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:48.446225 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.446218 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzvbg\" (UniqueName: \"kubernetes.io/projected/077ba44b-d69e-42f4-8511-b730c8c4855a-kube-api-access-fzvbg\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:48.446225 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.446229 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:48.446225 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.446238 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077ba44b-d69e-42f4-8511-b730c8c4855a-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:48.941032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.940993 4304 generic.go:356] "Generic (PLEG): container finished" podID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerID="15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803" exitCode=0 Nov 03 00:41:48.941230 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.941060 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" Nov 03 00:41:48.941230 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.941070 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" event={"ID":"077ba44b-d69e-42f4-8511-b730c8c4855a","Type":"ContainerDied","Data":"15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803"} Nov 03 00:41:48.941230 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.941144 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7" event={"ID":"077ba44b-d69e-42f4-8511-b730c8c4855a","Type":"ContainerDied","Data":"83c0d04187f6344e4aea3d592a81f08143fe899b687e28a78961c0e127573ad0"} Nov 03 00:41:48.941230 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.941176 4304 scope.go:122] "RemoveContainer" containerID="15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803" Nov 03 00:41:48.953225 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.953204 4304 scope.go:122] "RemoveContainer" containerID="2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd" Nov 03 00:41:48.962393 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.962369 4304 scope.go:122] "RemoveContainer" containerID="15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803" Nov 03 00:41:48.962675 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:41:48.962655 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803\": container with ID starting with 15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803 not found: ID does not exist" containerID="15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803" Nov 03 00:41:48.962755 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.962688 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803"} err="failed to get container status \"15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803\": rpc error: code = NotFound desc = could not find container \"15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803\": container with ID starting with 15bd2c857820d37ecce86ff7673590d6b5c7024e2ea15363bd240053d0630803 not found: ID does not exist" Nov 03 00:41:48.962755 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.962716 4304 scope.go:122] "RemoveContainer" containerID="2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd" Nov 03 00:41:48.962983 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:41:48.962957 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd\": container with ID starting with 2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd not found: ID does not exist" containerID="2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd" Nov 03 00:41:48.963093 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:48.962989 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd"} err="failed to get container status \"2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd\": rpc error: code = NotFound desc = could not find container \"2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd\": container with ID starting with 2ca2c7fba925664486cbf12b5584d986a850936b2a21bd58d70cded381673dbd not found: ID does not exist" Nov 03 00:41:49.110342 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:49.110260 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7"] Nov 03 00:41:49.114682 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:49.114646 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-0db2fb-openshift-default-6786b98dc7-gh8w7"] Nov 03 00:41:50.835263 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:41:50.835225 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" path="/var/lib/kubelet/pods/077ba44b-d69e-42f4-8511-b730c8c4855a/volumes" Nov 03 00:41:58.657352 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:41:58.657315 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:42:13.594993 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:13.594955 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6"] Nov 03 00:42:13.595484 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:13.595171 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" containerID="cri-o://19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea" gracePeriod=30 Nov 03 00:42:18.869007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.868951 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:42:18.924745 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.924719 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-envoy\") pod \"3765fa10-9d25-4251-b5c4-485fa784a0e1\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " Nov 03 00:42:18.924907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.924807 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-kube-api-access-qvgvp\" (UniqueName: \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-kube-api-access-qvgvp\") pod \"3765fa10-9d25-4251-b5c4-485fa784a0e1\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " Nov 03 00:42:18.924907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.924841 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-data\") pod \"3765fa10-9d25-4251-b5c4-485fa784a0e1\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " Nov 03 00:42:18.924984 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.924908 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3765fa10-9d25-4251-b5c4-485fa784a0e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3765fa10-9d25-4251-b5c4-485fa784a0e1-istiod-ca-cert\") pod \"3765fa10-9d25-4251-b5c4-485fa784a0e1\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " Nov 03 00:42:18.924984 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.924933 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-token\" (UniqueName: \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-token\") pod \"3765fa10-9d25-4251-b5c4-485fa784a0e1\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " Nov 03 00:42:18.924984 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.924954 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-socket\") pod \"3765fa10-9d25-4251-b5c4-485fa784a0e1\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " Nov 03 00:42:18.925145 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925043 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-certs\") pod \"3765fa10-9d25-4251-b5c4-485fa784a0e1\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " Nov 03 00:42:18.925145 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925085 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-podinfo\") pod \"3765fa10-9d25-4251-b5c4-485fa784a0e1\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " Nov 03 00:42:18.925145 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925113 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-credential-socket\") pod \"3765fa10-9d25-4251-b5c4-485fa784a0e1\" (UID: \"3765fa10-9d25-4251-b5c4-485fa784a0e1\") " Nov 03 00:42:18.925311 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925213 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-data" pod "3765fa10-9d25-4251-b5c4-485fa784a0e1" (UID: "3765fa10-9d25-4251-b5c4-485fa784a0e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:18.925311 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925288 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3765fa10-9d25-4251-b5c4-485fa784a0e1-istiod-ca-cert" pod "3765fa10-9d25-4251-b5c4-485fa784a0e1" (UID: "3765fa10-9d25-4251-b5c4-485fa784a0e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:18.925420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925302 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-socket" pod "3765fa10-9d25-4251-b5c4-485fa784a0e1" (UID: "3765fa10-9d25-4251-b5c4-485fa784a0e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:18.925475 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925425 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-certs" pod "3765fa10-9d25-4251-b5c4-485fa784a0e1" (UID: "3765fa10-9d25-4251-b5c4-485fa784a0e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:18.925526 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925500 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-credential-socket" pod "3765fa10-9d25-4251-b5c4-485fa784a0e1" (UID: "3765fa10-9d25-4251-b5c4-485fa784a0e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:18.925665 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925644 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3765fa10-9d25-4251-b5c4-485fa784a0e1-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:18.925755 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925671 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:18.925755 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925687 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:18.925755 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925700 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:18.925755 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.925715 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:18.927163 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.927139 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-envoy" pod "3765fa10-9d25-4251-b5c4-485fa784a0e1" (UID: "3765fa10-9d25-4251-b5c4-485fa784a0e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:18.927562 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.927542 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-podinfo" pod "3765fa10-9d25-4251-b5c4-485fa784a0e1" (UID: "3765fa10-9d25-4251-b5c4-485fa784a0e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:42:18.927640 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.927622 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-token" pod "3765fa10-9d25-4251-b5c4-485fa784a0e1" (UID: "3765fa10-9d25-4251-b5c4-485fa784a0e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:18.927682 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:18.927648 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-kube-api-access-qvgvp" pod "3765fa10-9d25-4251-b5c4-485fa784a0e1" (UID: "3765fa10-9d25-4251-b5c4-485fa784a0e1"). InnerVolumeSpecName "kube-api-access-qvgvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:19.027058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.027015 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvgvp\" (UniqueName: \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-kube-api-access-qvgvp\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.027058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.027056 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.027058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.027067 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.027287 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.027077 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3765fa10-9d25-4251-b5c4-485fa784a0e1-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.155642 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.155467 4304 generic.go:356] "Generic (PLEG): container finished" podID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerID="19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea" exitCode=0 Nov 03 00:42:19.155642 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.155559 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" Nov 03 00:42:19.156326 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.156075 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" event={"ID":"3765fa10-9d25-4251-b5c4-485fa784a0e1","Type":"ContainerDied","Data":"19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea"} Nov 03 00:42:19.156326 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.156173 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6" event={"ID":"3765fa10-9d25-4251-b5c4-485fa784a0e1","Type":"ContainerDied","Data":"f7c5b0b19676aab73990e6dc991b338bae3d1ad9e56641593667835926fdc18e"} Nov 03 00:42:19.156326 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.156203 4304 scope.go:122] "RemoveContainer" containerID="19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea" Nov 03 00:42:19.173394 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.173370 4304 scope.go:122] "RemoveContainer" containerID="c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b" Nov 03 00:42:19.198621 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.198592 4304 scope.go:122] "RemoveContainer" containerID="19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea" Nov 03 00:42:19.199492 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:19.199468 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea\": container with ID starting with 19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea not found: ID does not exist" containerID="19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea" Nov 03 00:42:19.199626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.199501 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea"} err="failed to get container status \"19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea\": rpc error: code = NotFound desc = could not find container \"19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea\": container with ID starting with 19e7cd57c3dbbf993e71f2d65e4b3376b18a1f42f3abc2821382bfe929d06cea not found: ID does not exist" Nov 03 00:42:19.199626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.199525 4304 scope.go:122] "RemoveContainer" containerID="c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b" Nov 03 00:42:19.199899 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:19.199846 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b\": container with ID starting with c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b not found: ID does not exist" containerID="c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b" Nov 03 00:42:19.199899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.199878 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b"} err="failed to get container status \"c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b\": rpc error: code = NotFound desc = could not find container \"c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b\": container with ID starting with c8366ba6d11092d6ba036d2352e806a2b53d5f8ad01569d54429f9ef5eab326b not found: ID does not exist" Nov 03 00:42:19.320522 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.320480 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6"] Nov 03 00:42:19.324571 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.324547 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-75dfee-openshift-default-5596f5b57-clwz6"] Nov 03 00:42:19.749932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.749896 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv"] Nov 03 00:42:19.860981 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:19.860950 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz"] Nov 03 00:42:20.829309 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:20.829275 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" path="/var/lib/kubelet/pods/3765fa10-9d25-4251-b5c4-485fa784a0e1/volumes" Nov 03 00:42:24.750961 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:24.750909 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" podUID="ade39df8-8a42-40ef-9652-1411ecdcb65f" containerName="payload-processing" containerID="cri-o://6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d" gracePeriod=25 Nov 03 00:42:24.861544 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:24.861488 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" podUID="9a1e9db6-48e7-4868-99d4-201349b1e2b9" containerName="payload-pre-processing" containerID="cri-o://fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed" gracePeriod=25 Nov 03 00:42:25.016768 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.016696 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:42:25.093032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.092991 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade39df8-8a42-40ef-9652-1411ecdcb65f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade39df8-8a42-40ef-9652-1411ecdcb65f-plugins-config-volume\") pod \"ade39df8-8a42-40ef-9652-1411ecdcb65f\" (UID: \"ade39df8-8a42-40ef-9652-1411ecdcb65f\") " Nov 03 00:42:25.093226 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.093207 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade39df8-8a42-40ef-9652-1411ecdcb65f-kube-api-access-4tvd8\" (UniqueName: \"kubernetes.io/projected/ade39df8-8a42-40ef-9652-1411ecdcb65f-kube-api-access-4tvd8\") pod \"ade39df8-8a42-40ef-9652-1411ecdcb65f\" (UID: \"ade39df8-8a42-40ef-9652-1411ecdcb65f\") " Nov 03 00:42:25.093402 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.093377 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade39df8-8a42-40ef-9652-1411ecdcb65f-plugins-config-volume" pod "ade39df8-8a42-40ef-9652-1411ecdcb65f" (UID: "ade39df8-8a42-40ef-9652-1411ecdcb65f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:25.093590 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.093575 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade39df8-8a42-40ef-9652-1411ecdcb65f-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:25.096873 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.096835 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade39df8-8a42-40ef-9652-1411ecdcb65f-kube-api-access-4tvd8" pod "ade39df8-8a42-40ef-9652-1411ecdcb65f" (UID: "ade39df8-8a42-40ef-9652-1411ecdcb65f"). InnerVolumeSpecName "kube-api-access-4tvd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:25.122800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.122757 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:42:25.194255 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.194226 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1e9db6-48e7-4868-99d4-201349b1e2b9-kube-api-access-2c874\" (UniqueName: \"kubernetes.io/projected/9a1e9db6-48e7-4868-99d4-201349b1e2b9-kube-api-access-2c874\") pod \"9a1e9db6-48e7-4868-99d4-201349b1e2b9\" (UID: \"9a1e9db6-48e7-4868-99d4-201349b1e2b9\") " Nov 03 00:42:25.194255 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.194252 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" event={"ID":"9a1e9db6-48e7-4868-99d4-201349b1e2b9","Type":"ContainerDied","Data":"fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed"} Nov 03 00:42:25.194515 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.194269 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1e9db6-48e7-4868-99d4-201349b1e2b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1e9db6-48e7-4868-99d4-201349b1e2b9-plugins-config-volume\") pod \"9a1e9db6-48e7-4868-99d4-201349b1e2b9\" (UID: \"9a1e9db6-48e7-4868-99d4-201349b1e2b9\") " Nov 03 00:42:25.194515 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.194280 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" Nov 03 00:42:25.194515 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.194305 4304 scope.go:122] "RemoveContainer" containerID="fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed" Nov 03 00:42:25.194515 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.194228 4304 generic.go:356] "Generic (PLEG): container finished" podID="9a1e9db6-48e7-4868-99d4-201349b1e2b9" containerID="fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed" exitCode=0 Nov 03 00:42:25.194515 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.194416 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz" event={"ID":"9a1e9db6-48e7-4868-99d4-201349b1e2b9","Type":"ContainerDied","Data":"7a030b6ae9ec7e53144fc5f930f35be00c2637d3bf1d4010f04e653786716f2c"} Nov 03 00:42:25.194717 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.194658 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tvd8\" (UniqueName: \"kubernetes.io/projected/ade39df8-8a42-40ef-9652-1411ecdcb65f-kube-api-access-4tvd8\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:25.194790 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.194746 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1e9db6-48e7-4868-99d4-201349b1e2b9-plugins-config-volume" pod "9a1e9db6-48e7-4868-99d4-201349b1e2b9" (UID: "9a1e9db6-48e7-4868-99d4-201349b1e2b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:25.196412 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.196387 4304 generic.go:356] "Generic (PLEG): container finished" podID="ade39df8-8a42-40ef-9652-1411ecdcb65f" containerID="6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d" exitCode=0 Nov 03 00:42:25.196524 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.196417 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" Nov 03 00:42:25.196524 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.196503 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" event={"ID":"ade39df8-8a42-40ef-9652-1411ecdcb65f","Type":"ContainerDied","Data":"6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d"} Nov 03 00:42:25.196653 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.196569 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv" event={"ID":"ade39df8-8a42-40ef-9652-1411ecdcb65f","Type":"ContainerDied","Data":"5524d441537a5f70320472b90f26b13a6bf689744451e5586261a04fb449e42d"} Nov 03 00:42:25.197128 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.197102 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1e9db6-48e7-4868-99d4-201349b1e2b9-kube-api-access-2c874" pod "9a1e9db6-48e7-4868-99d4-201349b1e2b9" (UID: "9a1e9db6-48e7-4868-99d4-201349b1e2b9"). InnerVolumeSpecName "kube-api-access-2c874". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:25.204977 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.204961 4304 scope.go:122] "RemoveContainer" containerID="fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed" Nov 03 00:42:25.205251 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:25.205229 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed\": container with ID starting with fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed not found: ID does not exist" containerID="fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed" Nov 03 00:42:25.205340 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.205258 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed"} err="failed to get container status \"fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed\": rpc error: code = NotFound desc = could not find container \"fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed\": container with ID starting with fed2a32b67fd2e30e3fd3d9c106d3bcecda80923e54357c2647f423e20a249ed not found: ID does not exist" Nov 03 00:42:25.205340 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.205277 4304 scope.go:122] "RemoveContainer" containerID="6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d" Nov 03 00:42:25.213740 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.213721 4304 scope.go:122] "RemoveContainer" containerID="6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d" Nov 03 00:42:25.214019 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:25.213998 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d\": container with ID starting with 6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d not found: ID does not exist" containerID="6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d" Nov 03 00:42:25.214094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.214024 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d"} err="failed to get container status \"6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d\": rpc error: code = NotFound desc = could not find container \"6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d\": container with ID starting with 6dc817733e472a781ce10cabb055d295cdf9a4add80571edabcec3b7464f132d not found: ID does not exist" Nov 03 00:42:25.296166 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.296081 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c874\" (UniqueName: \"kubernetes.io/projected/9a1e9db6-48e7-4868-99d4-201349b1e2b9-kube-api-access-2c874\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:25.296166 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.296106 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1e9db6-48e7-4868-99d4-201349b1e2b9-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:25.409968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.409936 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv"] Nov 03 00:42:25.412839 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.412813 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ada73-8569cf7f5-xqxhv"] Nov 03 00:42:25.614080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.613989 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz"] Nov 03 00:42:25.617672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:25.617646 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ada73-c595f55-gb8vz"] Nov 03 00:42:26.830440 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:26.830399 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1e9db6-48e7-4868-99d4-201349b1e2b9" path="/var/lib/kubelet/pods/9a1e9db6-48e7-4868-99d4-201349b1e2b9/volumes" Nov 03 00:42:26.831014 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:26.830991 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade39df8-8a42-40ef-9652-1411ecdcb65f" path="/var/lib/kubelet/pods/ade39df8-8a42-40ef-9652-1411ecdcb65f/volumes" Nov 03 00:42:28.655945 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:28.655901 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:42:28.795041 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:28.795002 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz"] Nov 03 00:42:28.795266 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:28.795244 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" podUID="e549a85b-87a0-495f-923c-3e817744f603" containerName="maas-api" containerID="cri-o://b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55" gracePeriod=30 Nov 03 00:42:29.076550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.076525 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:42:29.228669 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.228565 4304 generic.go:356] "Generic (PLEG): container finished" podID="e549a85b-87a0-495f-923c-3e817744f603" containerID="b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55" exitCode=0 Nov 03 00:42:29.228669 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.228631 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" event={"ID":"e549a85b-87a0-495f-923c-3e817744f603","Type":"ContainerDied","Data":"b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55"} Nov 03 00:42:29.229204 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.228701 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" event={"ID":"e549a85b-87a0-495f-923c-3e817744f603","Type":"ContainerDied","Data":"c03123769b1df4175f95db2badc47895f664c4b18522854ecb5592f89805e468"} Nov 03 00:42:29.229204 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.228718 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz" Nov 03 00:42:29.229204 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.228730 4304 scope.go:122] "RemoveContainer" containerID="b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55" Nov 03 00:42:29.244084 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.244056 4304 scope.go:122] "RemoveContainer" containerID="b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55" Nov 03 00:42:29.244246 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.244219 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-metrics-certs\") pod \"e549a85b-87a0-495f-923c-3e817744f603\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " Nov 03 00:42:29.244378 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.244337 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e549a85b-87a0-495f-923c-3e817744f603-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e549a85b-87a0-495f-923c-3e817744f603-openshift-service-ca\") pod \"e549a85b-87a0-495f-923c-3e817744f603\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " Nov 03 00:42:29.244443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.244382 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e549a85b-87a0-495f-923c-3e817744f603-kube-api-access-bt4lg\" (UniqueName: \"kubernetes.io/projected/e549a85b-87a0-495f-923c-3e817744f603-kube-api-access-bt4lg\") pod \"e549a85b-87a0-495f-923c-3e817744f603\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " Nov 03 00:42:29.244443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.244423 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls\") pod \"e549a85b-87a0-495f-923c-3e817744f603\" (UID: \"e549a85b-87a0-495f-923c-3e817744f603\") " Nov 03 00:42:29.244563 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:29.244470 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55\": container with ID starting with b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55 not found: ID does not exist" containerID="b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55" Nov 03 00:42:29.244563 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.244511 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55"} err="failed to get container status \"b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55\": rpc error: code = NotFound desc = could not find container \"b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55\": container with ID starting with b852c86a89050526727baedcc72286e029e5c88378d4ab6e2a048f3c209eaf55 not found: ID does not exist" Nov 03 00:42:29.244942 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.244884 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e549a85b-87a0-495f-923c-3e817744f603-openshift-service-ca" pod "e549a85b-87a0-495f-923c-3e817744f603" (UID: "e549a85b-87a0-495f-923c-3e817744f603"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:29.247018 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.246992 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64"] Nov 03 00:42:29.247266 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.247224 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" podUID="4902af84-c86b-46c7-a4f3-e0bac046ca77" containerName="payload-processing" containerID="cri-o://f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d" gracePeriod=30 Nov 03 00:42:29.248073 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.248054 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e549a85b-87a0-495f-923c-3e817744f603-kube-api-access-bt4lg" pod "e549a85b-87a0-495f-923c-3e817744f603" (UID: "e549a85b-87a0-495f-923c-3e817744f603"). InnerVolumeSpecName "kube-api-access-bt4lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:29.248159 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.248065 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls" pod "e549a85b-87a0-495f-923c-3e817744f603" (UID: "e549a85b-87a0-495f-923c-3e817744f603"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:29.253148 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.253118 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-metrics-certs" pod "e549a85b-87a0-495f-923c-3e817744f603" (UID: "e549a85b-87a0-495f-923c-3e817744f603"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:29.309619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.309588 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm"] Nov 03 00:42:29.309830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.309806 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" podUID="01f9ade9-8108-4870-a6ae-47007de2163f" containerName="payload-pre-processing" containerID="cri-o://d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366" gracePeriod=30 Nov 03 00:42:29.346089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.346060 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e549a85b-87a0-495f-923c-3e817744f603-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:29.346089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.346088 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt4lg\" (UniqueName: \"kubernetes.io/projected/e549a85b-87a0-495f-923c-3e817744f603-kube-api-access-bt4lg\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:29.346266 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.346103 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:29.346266 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.346114 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e549a85b-87a0-495f-923c-3e817744f603-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:29.563100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.563072 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:42:29.597109 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.597088 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:42:29.649729 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.649695 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4902af84-c86b-46c7-a4f3-e0bac046ca77-kube-api-access-ngfgs\" (UniqueName: \"kubernetes.io/projected/4902af84-c86b-46c7-a4f3-e0bac046ca77-kube-api-access-ngfgs\") pod \"4902af84-c86b-46c7-a4f3-e0bac046ca77\" (UID: \"4902af84-c86b-46c7-a4f3-e0bac046ca77\") " Nov 03 00:42:29.649987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.649959 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4902af84-c86b-46c7-a4f3-e0bac046ca77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4902af84-c86b-46c7-a4f3-e0bac046ca77-plugins-config-volume\") pod \"4902af84-c86b-46c7-a4f3-e0bac046ca77\" (UID: \"4902af84-c86b-46c7-a4f3-e0bac046ca77\") " Nov 03 00:42:29.650608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.650583 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4902af84-c86b-46c7-a4f3-e0bac046ca77-plugins-config-volume" pod "4902af84-c86b-46c7-a4f3-e0bac046ca77" (UID: "4902af84-c86b-46c7-a4f3-e0bac046ca77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:29.652250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.652225 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4902af84-c86b-46c7-a4f3-e0bac046ca77-kube-api-access-ngfgs" pod "4902af84-c86b-46c7-a4f3-e0bac046ca77" (UID: "4902af84-c86b-46c7-a4f3-e0bac046ca77"). InnerVolumeSpecName "kube-api-access-ngfgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:29.653879 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.653857 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz"] Nov 03 00:42:29.658905 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.658883 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-011885-7985786468-z4rxz"] Nov 03 00:42:29.751542 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.751466 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f9ade9-8108-4870-a6ae-47007de2163f-kube-api-access-jnt7s\" (UniqueName: \"kubernetes.io/projected/01f9ade9-8108-4870-a6ae-47007de2163f-kube-api-access-jnt7s\") pod \"01f9ade9-8108-4870-a6ae-47007de2163f\" (UID: \"01f9ade9-8108-4870-a6ae-47007de2163f\") " Nov 03 00:42:29.751699 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.751603 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f9ade9-8108-4870-a6ae-47007de2163f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f9ade9-8108-4870-a6ae-47007de2163f-plugins-config-volume\") pod \"01f9ade9-8108-4870-a6ae-47007de2163f\" (UID: \"01f9ade9-8108-4870-a6ae-47007de2163f\") " Nov 03 00:42:29.751972 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.751940 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f9ade9-8108-4870-a6ae-47007de2163f-plugins-config-volume" pod "01f9ade9-8108-4870-a6ae-47007de2163f" (UID: "01f9ade9-8108-4870-a6ae-47007de2163f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:29.752086 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.752072 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4902af84-c86b-46c7-a4f3-e0bac046ca77-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:29.752136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.752088 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f9ade9-8108-4870-a6ae-47007de2163f-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:29.752136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.752097 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngfgs\" (UniqueName: \"kubernetes.io/projected/4902af84-c86b-46c7-a4f3-e0bac046ca77-kube-api-access-ngfgs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:29.754163 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.754139 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f9ade9-8108-4870-a6ae-47007de2163f-kube-api-access-jnt7s" pod "01f9ade9-8108-4870-a6ae-47007de2163f" (UID: "01f9ade9-8108-4870-a6ae-47007de2163f"). InnerVolumeSpecName "kube-api-access-jnt7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:29.853209 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:29.853173 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnt7s\" (UniqueName: \"kubernetes.io/projected/01f9ade9-8108-4870-a6ae-47007de2163f-kube-api-access-jnt7s\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:30.236386 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.236352 4304 generic.go:356] "Generic (PLEG): container finished" podID="01f9ade9-8108-4870-a6ae-47007de2163f" containerID="d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366" exitCode=0 Nov 03 00:42:30.236573 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.236396 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" Nov 03 00:42:30.236573 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.236436 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" event={"ID":"01f9ade9-8108-4870-a6ae-47007de2163f","Type":"ContainerDied","Data":"d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366"} Nov 03 00:42:30.236573 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.236492 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm" event={"ID":"01f9ade9-8108-4870-a6ae-47007de2163f","Type":"ContainerDied","Data":"dd2deeaf1f5d314256bfe0047eb6d1e46e2552533fc3364f1a3ae25c059eadd4"} Nov 03 00:42:30.236573 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.236528 4304 scope.go:122] "RemoveContainer" containerID="d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366" Nov 03 00:42:30.239092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.239071 4304 generic.go:356] "Generic (PLEG): container finished" podID="4902af84-c86b-46c7-a4f3-e0bac046ca77" containerID="f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d" exitCode=0 Nov 03 00:42:30.239220 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.239105 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" event={"ID":"4902af84-c86b-46c7-a4f3-e0bac046ca77","Type":"ContainerDied","Data":"f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d"} Nov 03 00:42:30.239220 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.239168 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" event={"ID":"4902af84-c86b-46c7-a4f3-e0bac046ca77","Type":"ContainerDied","Data":"298feabffa76bf68e27fc3c70a03ad8e1e867c347bfe6166ca68562ef80956b0"} Nov 03 00:42:30.239220 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.239128 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64" Nov 03 00:42:30.247162 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.247147 4304 scope.go:122] "RemoveContainer" containerID="d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366" Nov 03 00:42:30.247428 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:30.247408 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366\": container with ID starting with d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366 not found: ID does not exist" containerID="d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366" Nov 03 00:42:30.247499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.247435 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366"} err="failed to get container status \"d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366\": rpc error: code = NotFound desc = could not find container \"d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366\": container with ID starting with d2c315b2f803b608cab1fbf02a6a2849ca8a2d1432e1a28f6f854b5bdc3fa366 not found: ID does not exist" Nov 03 00:42:30.247499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.247453 4304 scope.go:122] "RemoveContainer" containerID="f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d" Nov 03 00:42:30.256163 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.256148 4304 scope.go:122] "RemoveContainer" containerID="f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d" Nov 03 00:42:30.256456 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:30.256437 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d\": container with ID starting with f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d not found: ID does not exist" containerID="f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d" Nov 03 00:42:30.256537 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.256461 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d"} err="failed to get container status \"f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d\": rpc error: code = NotFound desc = could not find container \"f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d\": container with ID starting with f1dfe370b7ed1b9a45fd2114b8b9464bb7ed6204eea2e8eec61ca14fef58c16d not found: ID does not exist" Nov 03 00:42:30.403386 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.403358 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm"] Nov 03 00:42:30.405916 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.405898 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6f64887675-2kmfm"] Nov 03 00:42:30.512589 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.512515 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64"] Nov 03 00:42:30.518514 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.518489 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-644fff48b-n7z64"] Nov 03 00:42:30.830039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.829971 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f9ade9-8108-4870-a6ae-47007de2163f" path="/var/lib/kubelet/pods/01f9ade9-8108-4870-a6ae-47007de2163f/volumes" Nov 03 00:42:30.830416 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.830402 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4902af84-c86b-46c7-a4f3-e0bac046ca77" path="/var/lib/kubelet/pods/4902af84-c86b-46c7-a4f3-e0bac046ca77/volumes" Nov 03 00:42:30.830826 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:30.830812 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e549a85b-87a0-495f-923c-3e817744f603" path="/var/lib/kubelet/pods/e549a85b-87a0-495f-923c-3e817744f603/volumes" Nov 03 00:42:41.533296 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:41.533261 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c"] Nov 03 00:42:41.533661 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:41.533484 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" containerID="cri-o://623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66" gracePeriod=30 Nov 03 00:42:42.223507 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:42.223472 4304 patch_prober.go:29] interesting pod/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:42:42.223801 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:42.223749 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:42:43.630744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:43.630708 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq"] Nov 03 00:42:43.631215 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:43.630927 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerName="istio-proxy" containerID="cri-o://b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6" gracePeriod=30 Nov 03 00:42:46.801388 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.801332 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:42:46.860031 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.859999 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-credential-socket\") pod \"842da41d-8b5e-4197-9b92-5fa6e4d57562\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " Nov 03 00:42:46.860175 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860036 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-token\" (UniqueName: \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-token\") pod \"842da41d-8b5e-4197-9b92-5fa6e4d57562\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " Nov 03 00:42:46.860175 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860072 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842da41d-8b5e-4197-9b92-5fa6e4d57562-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842da41d-8b5e-4197-9b92-5fa6e4d57562-istiod-ca-cert\") pod \"842da41d-8b5e-4197-9b92-5fa6e4d57562\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " Nov 03 00:42:46.860175 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860106 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-certs\") pod \"842da41d-8b5e-4197-9b92-5fa6e4d57562\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " Nov 03 00:42:46.860175 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860173 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-data\") pod \"842da41d-8b5e-4197-9b92-5fa6e4d57562\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " Nov 03 00:42:46.860410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860190 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-kube-api-access-glpqx\" (UniqueName: \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-kube-api-access-glpqx\") pod \"842da41d-8b5e-4197-9b92-5fa6e4d57562\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " Nov 03 00:42:46.860410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860246 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-envoy\") pod \"842da41d-8b5e-4197-9b92-5fa6e4d57562\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " Nov 03 00:42:46.860410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860297 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-podinfo\") pod \"842da41d-8b5e-4197-9b92-5fa6e4d57562\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " Nov 03 00:42:46.860410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860302 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-credential-socket" pod "842da41d-8b5e-4197-9b92-5fa6e4d57562" (UID: "842da41d-8b5e-4197-9b92-5fa6e4d57562"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:46.860410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860384 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-socket\") pod \"842da41d-8b5e-4197-9b92-5fa6e4d57562\" (UID: \"842da41d-8b5e-4197-9b92-5fa6e4d57562\") " Nov 03 00:42:46.860663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860435 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-certs" pod "842da41d-8b5e-4197-9b92-5fa6e4d57562" (UID: "842da41d-8b5e-4197-9b92-5fa6e4d57562"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:46.860663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860450 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-data" pod "842da41d-8b5e-4197-9b92-5fa6e4d57562" (UID: "842da41d-8b5e-4197-9b92-5fa6e4d57562"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:46.860663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860517 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842da41d-8b5e-4197-9b92-5fa6e4d57562-istiod-ca-cert" pod "842da41d-8b5e-4197-9b92-5fa6e4d57562" (UID: "842da41d-8b5e-4197-9b92-5fa6e4d57562"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:46.860860 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860719 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-socket" pod "842da41d-8b5e-4197-9b92-5fa6e4d57562" (UID: "842da41d-8b5e-4197-9b92-5fa6e4d57562"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:46.860860 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860856 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842da41d-8b5e-4197-9b92-5fa6e4d57562-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:46.861075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860873 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:46.861075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860887 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:46.861075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860900 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:46.861075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.860914 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:46.862759 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.862731 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-envoy" pod "842da41d-8b5e-4197-9b92-5fa6e4d57562" (UID: "842da41d-8b5e-4197-9b92-5fa6e4d57562"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:46.862759 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.862742 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-podinfo" pod "842da41d-8b5e-4197-9b92-5fa6e4d57562" (UID: "842da41d-8b5e-4197-9b92-5fa6e4d57562"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:42:46.863066 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.863046 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-kube-api-access-glpqx" pod "842da41d-8b5e-4197-9b92-5fa6e4d57562" (UID: "842da41d-8b5e-4197-9b92-5fa6e4d57562"). InnerVolumeSpecName "kube-api-access-glpqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:46.863170 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.863073 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-token" pod "842da41d-8b5e-4197-9b92-5fa6e4d57562" (UID: "842da41d-8b5e-4197-9b92-5fa6e4d57562"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:46.961538 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.961509 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glpqx\" (UniqueName: \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-kube-api-access-glpqx\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:46.961538 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.961535 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:46.961721 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.961544 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:46.961721 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:46.961554 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842da41d-8b5e-4197-9b92-5fa6e4d57562-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:47.343855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.343820 4304 generic.go:356] "Generic (PLEG): container finished" podID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerID="623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66" exitCode=0 Nov 03 00:42:47.344022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.343871 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" Nov 03 00:42:47.344022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.343904 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" event={"ID":"842da41d-8b5e-4197-9b92-5fa6e4d57562","Type":"ContainerDied","Data":"623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66"} Nov 03 00:42:47.344022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.343961 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c" event={"ID":"842da41d-8b5e-4197-9b92-5fa6e4d57562","Type":"ContainerDied","Data":"f85a6b35e4c1a2306ada6a11c6043e340a3040686bdb9a1f20674d32e2959567"} Nov 03 00:42:47.344022 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.343983 4304 scope.go:122] "RemoveContainer" containerID="623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66" Nov 03 00:42:47.354407 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.354390 4304 scope.go:122] "RemoveContainer" containerID="e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f" Nov 03 00:42:47.362701 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.362686 4304 scope.go:122] "RemoveContainer" containerID="623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66" Nov 03 00:42:47.363107 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:47.363090 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66\": container with ID starting with 623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66 not found: ID does not exist" containerID="623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66" Nov 03 00:42:47.363183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.363115 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66"} err="failed to get container status \"623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66\": rpc error: code = NotFound desc = could not find container \"623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66\": container with ID starting with 623c7273d4a0af2385fc92947efc42d839d887252bd23ac05c759879315bff66 not found: ID does not exist" Nov 03 00:42:47.363183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.363138 4304 scope.go:122] "RemoveContainer" containerID="e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f" Nov 03 00:42:47.363392 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:47.363374 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f\": container with ID starting with e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f not found: ID does not exist" containerID="e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f" Nov 03 00:42:47.363441 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.363400 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f"} err="failed to get container status \"e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f\": rpc error: code = NotFound desc = could not find container \"e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f\": container with ID starting with e37dac7b6ab33eb32a79ab32fd5b6709bba2cc6b86836c60012b179aa469c98f not found: ID does not exist" Nov 03 00:42:47.510612 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.510574 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c"] Nov 03 00:42:47.559918 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:47.559885 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-011885-openshift-default-9c64588c5-88z8c"] Nov 03 00:42:48.831857 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.831817 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" path="/var/lib/kubelet/pods/842da41d-8b5e-4197-9b92-5fa6e4d57562/volumes" Nov 03 00:42:48.885095 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.885070 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:42:48.985759 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.985688 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90a54d0-dd85-433d-a372-fecd7d8c5106-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90a54d0-dd85-433d-a372-fecd7d8c5106-istiod-ca-cert\") pod \"b90a54d0-dd85-433d-a372-fecd7d8c5106\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " Nov 03 00:42:48.985930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.985802 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-socket\") pod \"b90a54d0-dd85-433d-a372-fecd7d8c5106\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " Nov 03 00:42:48.985930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.985861 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-kube-api-access-86p9x\" (UniqueName: \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-kube-api-access-86p9x\") pod \"b90a54d0-dd85-433d-a372-fecd7d8c5106\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " Nov 03 00:42:48.985930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.985889 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-envoy\") pod \"b90a54d0-dd85-433d-a372-fecd7d8c5106\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " Nov 03 00:42:48.985930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.985917 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-credential-socket\") pod \"b90a54d0-dd85-433d-a372-fecd7d8c5106\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " Nov 03 00:42:48.986205 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.985991 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-podinfo\") pod \"b90a54d0-dd85-433d-a372-fecd7d8c5106\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " Nov 03 00:42:48.986205 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.986085 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-token\" (UniqueName: \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-token\") pod \"b90a54d0-dd85-433d-a372-fecd7d8c5106\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " Nov 03 00:42:48.986205 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.986114 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-certs\") pod \"b90a54d0-dd85-433d-a372-fecd7d8c5106\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " Nov 03 00:42:48.986205 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.986115 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-socket" pod "b90a54d0-dd85-433d-a372-fecd7d8c5106" (UID: "b90a54d0-dd85-433d-a372-fecd7d8c5106"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:48.986205 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.986134 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90a54d0-dd85-433d-a372-fecd7d8c5106-istiod-ca-cert" pod "b90a54d0-dd85-433d-a372-fecd7d8c5106" (UID: "b90a54d0-dd85-433d-a372-fecd7d8c5106"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:48.986205 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.986169 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-data\") pod \"b90a54d0-dd85-433d-a372-fecd7d8c5106\" (UID: \"b90a54d0-dd85-433d-a372-fecd7d8c5106\") " Nov 03 00:42:48.986583 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.986287 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-credential-socket" pod "b90a54d0-dd85-433d-a372-fecd7d8c5106" (UID: "b90a54d0-dd85-433d-a372-fecd7d8c5106"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:48.987316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.986644 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90a54d0-dd85-433d-a372-fecd7d8c5106-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:48.987316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.986800 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:48.987316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.986815 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:48.987316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.987057 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-certs" pod "b90a54d0-dd85-433d-a372-fecd7d8c5106" (UID: "b90a54d0-dd85-433d-a372-fecd7d8c5106"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:48.987316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.987069 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-data" pod "b90a54d0-dd85-433d-a372-fecd7d8c5106" (UID: "b90a54d0-dd85-433d-a372-fecd7d8c5106"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:48.988588 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.988562 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-podinfo" pod "b90a54d0-dd85-433d-a372-fecd7d8c5106" (UID: "b90a54d0-dd85-433d-a372-fecd7d8c5106"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:42:48.988588 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.988577 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-envoy" pod "b90a54d0-dd85-433d-a372-fecd7d8c5106" (UID: "b90a54d0-dd85-433d-a372-fecd7d8c5106"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:48.988734 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.988563 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-kube-api-access-86p9x" pod "b90a54d0-dd85-433d-a372-fecd7d8c5106" (UID: "b90a54d0-dd85-433d-a372-fecd7d8c5106"). InnerVolumeSpecName "kube-api-access-86p9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:48.989225 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:48.989203 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-token" pod "b90a54d0-dd85-433d-a372-fecd7d8c5106" (UID: "b90a54d0-dd85-433d-a372-fecd7d8c5106"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:49.088253 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.088211 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.088253 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.088245 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.088253 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.088258 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.088499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.088270 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.088499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.088283 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86p9x\" (UniqueName: \"kubernetes.io/projected/b90a54d0-dd85-433d-a372-fecd7d8c5106-kube-api-access-86p9x\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.088499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.088296 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90a54d0-dd85-433d-a372-fecd7d8c5106-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.358557 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.358466 4304 generic.go:356] "Generic (PLEG): container finished" podID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerID="b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6" exitCode=0 Nov 03 00:42:49.358557 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.358518 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" event={"ID":"b90a54d0-dd85-433d-a372-fecd7d8c5106","Type":"ContainerDied","Data":"b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6"} Nov 03 00:42:49.358748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.358561 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" event={"ID":"b90a54d0-dd85-433d-a372-fecd7d8c5106","Type":"ContainerDied","Data":"138ee7c2a506cdecb11677a048276653a9a1647327d7b4ae74d27bab37ba8c1b"} Nov 03 00:42:49.358748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.358582 4304 scope.go:122] "RemoveContainer" containerID="b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6" Nov 03 00:42:49.358748 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.358523 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq" Nov 03 00:42:49.368828 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.368802 4304 scope.go:122] "RemoveContainer" containerID="b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6" Nov 03 00:42:49.369157 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:49.369135 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6\": container with ID starting with b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6 not found: ID does not exist" containerID="b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6" Nov 03 00:42:49.369255 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.369166 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6"} err="failed to get container status \"b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6\": rpc error: code = NotFound desc = could not find container \"b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6\": container with ID starting with b6324b81ab39e5a371c0bce7dfabba61a9c738db0bad75b48a5fe677ecc494f6 not found: ID does not exist" Nov 03 00:42:49.522626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.522594 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq"] Nov 03 00:42:49.573677 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:49.573648 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjhqq"] Nov 03 00:42:50.828988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:42:50.828953 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" path="/var/lib/kubelet/pods/b90a54d0-dd85-433d-a372-fecd7d8c5106/volumes" Nov 03 00:42:58.654769 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:42:58.654729 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:43:05.267814 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:05.267738 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2"] Nov 03 00:43:09.440377 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.440344 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87"] Nov 03 00:43:09.440812 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.440547 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" podUID="5212a935-d625-48fd-a814-171f2cf68758" containerName="maas-api" containerID="cri-o://3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9" gracePeriod=30 Nov 03 00:43:09.744564 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.744536 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:43:09.803741 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.803709 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5212a935-d625-48fd-a814-171f2cf68758-kube-api-access-wxrcl\" (UniqueName: \"kubernetes.io/projected/5212a935-d625-48fd-a814-171f2cf68758-kube-api-access-wxrcl\") pod \"5212a935-d625-48fd-a814-171f2cf68758\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " Nov 03 00:43:09.803964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.803790 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-maas-api-tls\") pod \"5212a935-d625-48fd-a814-171f2cf68758\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " Nov 03 00:43:09.803964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.803889 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-metrics-certs\") pod \"5212a935-d625-48fd-a814-171f2cf68758\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " Nov 03 00:43:09.804095 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.804044 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5212a935-d625-48fd-a814-171f2cf68758-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5212a935-d625-48fd-a814-171f2cf68758-openshift-service-ca\") pod \"5212a935-d625-48fd-a814-171f2cf68758\" (UID: \"5212a935-d625-48fd-a814-171f2cf68758\") " Nov 03 00:43:09.804606 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.804450 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5212a935-d625-48fd-a814-171f2cf68758-openshift-service-ca" pod "5212a935-d625-48fd-a814-171f2cf68758" (UID: "5212a935-d625-48fd-a814-171f2cf68758"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:09.806115 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.806080 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-metrics-certs" pod "5212a935-d625-48fd-a814-171f2cf68758" (UID: "5212a935-d625-48fd-a814-171f2cf68758"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:09.806221 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.806109 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-maas-api-tls" pod "5212a935-d625-48fd-a814-171f2cf68758" (UID: "5212a935-d625-48fd-a814-171f2cf68758"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:09.806521 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.806495 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5212a935-d625-48fd-a814-171f2cf68758-kube-api-access-wxrcl" pod "5212a935-d625-48fd-a814-171f2cf68758" (UID: "5212a935-d625-48fd-a814-171f2cf68758"). InnerVolumeSpecName "kube-api-access-wxrcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:09.905667 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.905582 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxrcl\" (UniqueName: \"kubernetes.io/projected/5212a935-d625-48fd-a814-171f2cf68758-kube-api-access-wxrcl\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:09.905667 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.905615 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:09.905667 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.905630 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5212a935-d625-48fd-a814-171f2cf68758-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:09.905667 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:09.905643 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5212a935-d625-48fd-a814-171f2cf68758-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:10.268568 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.268506 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" podUID="0692bf41-be30-48ae-9a89-8e0989352a36" containerName="payload-pre-processing" containerID="cri-o://ed901658cbb61439ad274b3c2ff926e9dfc8ee593f4ecd647a45e483a1b0a18e" gracePeriod=25 Nov 03 00:43:10.495002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.490610 4304 generic.go:356] "Generic (PLEG): container finished" podID="5212a935-d625-48fd-a814-171f2cf68758" containerID="3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9" exitCode=0 Nov 03 00:43:10.495002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.490737 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" event={"ID":"5212a935-d625-48fd-a814-171f2cf68758","Type":"ContainerDied","Data":"3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9"} Nov 03 00:43:10.495002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.490802 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" event={"ID":"5212a935-d625-48fd-a814-171f2cf68758","Type":"ContainerDied","Data":"5a8956a97acc3b2f8cd0c0e46678c29c29cd1f3e2e5119f6da01e7ebc2bb8bf2"} Nov 03 00:43:10.495002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.490824 4304 scope.go:122] "RemoveContainer" containerID="3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9" Nov 03 00:43:10.495002 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.490941 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87" Nov 03 00:43:10.498821 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.498044 4304 generic.go:356] "Generic (PLEG): container finished" podID="0692bf41-be30-48ae-9a89-8e0989352a36" containerID="ed901658cbb61439ad274b3c2ff926e9dfc8ee593f4ecd647a45e483a1b0a18e" exitCode=0 Nov 03 00:43:10.498821 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.498133 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" event={"ID":"0692bf41-be30-48ae-9a89-8e0989352a36","Type":"ContainerDied","Data":"ed901658cbb61439ad274b3c2ff926e9dfc8ee593f4ecd647a45e483a1b0a18e"} Nov 03 00:43:10.513005 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.509842 4304 scope.go:122] "RemoveContainer" containerID="3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9" Nov 03 00:43:10.513005 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:43:10.510115 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9\": container with ID starting with 3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9 not found: ID does not exist" containerID="3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9" Nov 03 00:43:10.513005 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.510143 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9"} err="failed to get container status \"3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9\": rpc error: code = NotFound desc = could not find container \"3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9\": container with ID starting with 3b797930dcf7e782990654ba3af7e39d9105ed8896b016cf845edc6b3acbb3c9 not found: ID does not exist" Nov 03 00:43:10.589623 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.589597 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:43:10.623245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.622445 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0692bf41-be30-48ae-9a89-8e0989352a36-kube-api-access-vpjgt\" (UniqueName: \"kubernetes.io/projected/0692bf41-be30-48ae-9a89-8e0989352a36-kube-api-access-vpjgt\") pod \"0692bf41-be30-48ae-9a89-8e0989352a36\" (UID: \"0692bf41-be30-48ae-9a89-8e0989352a36\") " Nov 03 00:43:10.623245 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.622559 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0692bf41-be30-48ae-9a89-8e0989352a36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0692bf41-be30-48ae-9a89-8e0989352a36-plugins-config-volume\") pod \"0692bf41-be30-48ae-9a89-8e0989352a36\" (UID: \"0692bf41-be30-48ae-9a89-8e0989352a36\") " Nov 03 00:43:10.623922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.623585 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0692bf41-be30-48ae-9a89-8e0989352a36-plugins-config-volume" pod "0692bf41-be30-48ae-9a89-8e0989352a36" (UID: "0692bf41-be30-48ae-9a89-8e0989352a36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:10.627936 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.627905 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0692bf41-be30-48ae-9a89-8e0989352a36-kube-api-access-vpjgt" pod "0692bf41-be30-48ae-9a89-8e0989352a36" (UID: "0692bf41-be30-48ae-9a89-8e0989352a36"). InnerVolumeSpecName "kube-api-access-vpjgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:10.674152 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.674122 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87"] Nov 03 00:43:10.722805 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.722761 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ada73-775dd97d76-55m87"] Nov 03 00:43:10.723804 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.723762 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0692bf41-be30-48ae-9a89-8e0989352a36-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:10.723863 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.723815 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpjgt\" (UniqueName: \"kubernetes.io/projected/0692bf41-be30-48ae-9a89-8e0989352a36-kube-api-access-vpjgt\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:10.832351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:10.832264 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5212a935-d625-48fd-a814-171f2cf68758" path="/var/lib/kubelet/pods/5212a935-d625-48fd-a814-171f2cf68758/volumes" Nov 03 00:43:11.506011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:11.505981 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" Nov 03 00:43:11.506011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:11.506009 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2" event={"ID":"0692bf41-be30-48ae-9a89-8e0989352a36","Type":"ContainerDied","Data":"df8cbcc2a92062d1d15dd69967e9f0477f19ffefd6663210d779ece793d5779b"} Nov 03 00:43:11.506485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:11.506063 4304 scope.go:122] "RemoveContainer" containerID="ed901658cbb61439ad274b3c2ff926e9dfc8ee593f4ecd647a45e483a1b0a18e" Nov 03 00:43:11.668365 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:11.668328 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2"] Nov 03 00:43:11.672177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:11.672152 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ff10d6-55f4674d75-4m2w2"] Nov 03 00:43:12.829261 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:12.829225 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0692bf41-be30-48ae-9a89-8e0989352a36" path="/var/lib/kubelet/pods/0692bf41-be30-48ae-9a89-8e0989352a36/volumes" Nov 03 00:43:17.559455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:17.559417 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4"] Nov 03 00:43:17.560015 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:17.559676 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" containerID="cri-o://0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab" gracePeriod=30 Nov 03 00:43:18.829977 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:18.829943 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h"] Nov 03 00:43:18.830412 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:18.830184 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" containerID="cri-o://67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd" gracePeriod=30 Nov 03 00:43:20.501769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.501740 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2"] Nov 03 00:43:20.502326 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.501957 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" podUID="bef67aae-a041-4563-866c-e803dde848fd" containerName="maas-api" containerID="cri-o://92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777" gracePeriod=30 Nov 03 00:43:20.759417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.759360 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:43:20.830510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.830482 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-maas-api-tls\") pod \"bef67aae-a041-4563-866c-e803dde848fd\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " Nov 03 00:43:20.830680 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.830538 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef67aae-a041-4563-866c-e803dde848fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef67aae-a041-4563-866c-e803dde848fd-openshift-service-ca\") pod \"bef67aae-a041-4563-866c-e803dde848fd\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " Nov 03 00:43:20.830763 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.830691 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef67aae-a041-4563-866c-e803dde848fd-kube-api-access-b7fcj\" (UniqueName: \"kubernetes.io/projected/bef67aae-a041-4563-866c-e803dde848fd-kube-api-access-b7fcj\") pod \"bef67aae-a041-4563-866c-e803dde848fd\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " Nov 03 00:43:20.830843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.830826 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-metrics-certs\") pod \"bef67aae-a041-4563-866c-e803dde848fd\" (UID: \"bef67aae-a041-4563-866c-e803dde848fd\") " Nov 03 00:43:20.830900 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.830880 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef67aae-a041-4563-866c-e803dde848fd-openshift-service-ca" pod "bef67aae-a041-4563-866c-e803dde848fd" (UID: "bef67aae-a041-4563-866c-e803dde848fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:20.831138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.831118 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef67aae-a041-4563-866c-e803dde848fd-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:20.832672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.832651 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-maas-api-tls" pod "bef67aae-a041-4563-866c-e803dde848fd" (UID: "bef67aae-a041-4563-866c-e803dde848fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:20.832896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.832875 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-metrics-certs" pod "bef67aae-a041-4563-866c-e803dde848fd" (UID: "bef67aae-a041-4563-866c-e803dde848fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:20.833012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.832996 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef67aae-a041-4563-866c-e803dde848fd-kube-api-access-b7fcj" pod "bef67aae-a041-4563-866c-e803dde848fd" (UID: "bef67aae-a041-4563-866c-e803dde848fd"). InnerVolumeSpecName "kube-api-access-b7fcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:20.932446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.932413 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7fcj\" (UniqueName: \"kubernetes.io/projected/bef67aae-a041-4563-866c-e803dde848fd-kube-api-access-b7fcj\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:20.932446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.932442 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:20.932446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:20.932452 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bef67aae-a041-4563-866c-e803dde848fd-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:21.575501 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:21.575468 4304 generic.go:356] "Generic (PLEG): container finished" podID="bef67aae-a041-4563-866c-e803dde848fd" containerID="92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777" exitCode=0 Nov 03 00:43:21.575967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:21.575512 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" event={"ID":"bef67aae-a041-4563-866c-e803dde848fd","Type":"ContainerDied","Data":"92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777"} Nov 03 00:43:21.575967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:21.575533 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" Nov 03 00:43:21.575967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:21.575560 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2" event={"ID":"bef67aae-a041-4563-866c-e803dde848fd","Type":"ContainerDied","Data":"ad779c04e263b2c1d703a3b95b0e3e5f8b1c097a7dd866f1e5fbe14906698bb3"} Nov 03 00:43:21.575967 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:21.575584 4304 scope.go:122] "RemoveContainer" containerID="92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777" Nov 03 00:43:21.585998 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:21.585975 4304 scope.go:122] "RemoveContainer" containerID="92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777" Nov 03 00:43:21.586273 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:43:21.586252 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777\": container with ID starting with 92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777 not found: ID does not exist" containerID="92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777" Nov 03 00:43:21.586332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:21.586282 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777"} err="failed to get container status \"92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777\": rpc error: code = NotFound desc = could not find container \"92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777\": container with ID starting with 92f107d1c4904cf2738589446727acf12488c25932801193e5326d105e56d777 not found: ID does not exist" Nov 03 00:43:21.740009 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:21.739973 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2"] Nov 03 00:43:21.789439 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:21.789409 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ff10d6-6779df465f-tspv2"] Nov 03 00:43:22.822170 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.822140 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:43:22.836249 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.829688 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef67aae-a041-4563-866c-e803dde848fd" path="/var/lib/kubelet/pods/bef67aae-a041-4563-866c-e803dde848fd/volumes" Nov 03 00:43:22.951694 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.951660 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dabb012-d842-4c6b-8937-de5b400dbc46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dabb012-d842-4c6b-8937-de5b400dbc46-istiod-ca-cert\") pod \"6dabb012-d842-4c6b-8937-de5b400dbc46\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " Nov 03 00:43:22.951902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.951714 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-podinfo\") pod \"6dabb012-d842-4c6b-8937-de5b400dbc46\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " Nov 03 00:43:22.951902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.951768 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-socket\") pod \"6dabb012-d842-4c6b-8937-de5b400dbc46\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " Nov 03 00:43:22.951902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.951817 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-certs\") pod \"6dabb012-d842-4c6b-8937-de5b400dbc46\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " Nov 03 00:43:22.951902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.951861 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-kube-api-access-zjlhw\" (UniqueName: \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-kube-api-access-zjlhw\") pod \"6dabb012-d842-4c6b-8937-de5b400dbc46\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " Nov 03 00:43:22.951902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.951877 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-token\" (UniqueName: \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-token\") pod \"6dabb012-d842-4c6b-8937-de5b400dbc46\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " Nov 03 00:43:22.952184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.951931 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-credential-socket\") pod \"6dabb012-d842-4c6b-8937-de5b400dbc46\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " Nov 03 00:43:22.952184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.951973 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-data\") pod \"6dabb012-d842-4c6b-8937-de5b400dbc46\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " Nov 03 00:43:22.952184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.952004 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-envoy\") pod \"6dabb012-d842-4c6b-8937-de5b400dbc46\" (UID: \"6dabb012-d842-4c6b-8937-de5b400dbc46\") " Nov 03 00:43:22.952184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.952132 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-certs" pod "6dabb012-d842-4c6b-8937-de5b400dbc46" (UID: "6dabb012-d842-4c6b-8937-de5b400dbc46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:22.952184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.952048 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dabb012-d842-4c6b-8937-de5b400dbc46-istiod-ca-cert" pod "6dabb012-d842-4c6b-8937-de5b400dbc46" (UID: "6dabb012-d842-4c6b-8937-de5b400dbc46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:22.952184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.952148 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-socket" pod "6dabb012-d842-4c6b-8937-de5b400dbc46" (UID: "6dabb012-d842-4c6b-8937-de5b400dbc46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:22.952715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.952371 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-credential-socket" pod "6dabb012-d842-4c6b-8937-de5b400dbc46" (UID: "6dabb012-d842-4c6b-8937-de5b400dbc46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:22.952715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.952423 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:22.952715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.952441 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:22.952715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.952455 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dabb012-d842-4c6b-8937-de5b400dbc46-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:22.952715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.952548 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-data" pod "6dabb012-d842-4c6b-8937-de5b400dbc46" (UID: "6dabb012-d842-4c6b-8937-de5b400dbc46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:22.954303 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.954280 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-podinfo" pod "6dabb012-d842-4c6b-8937-de5b400dbc46" (UID: "6dabb012-d842-4c6b-8937-de5b400dbc46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:43:22.954453 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.954437 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-kube-api-access-zjlhw" pod "6dabb012-d842-4c6b-8937-de5b400dbc46" (UID: "6dabb012-d842-4c6b-8937-de5b400dbc46"). InnerVolumeSpecName "kube-api-access-zjlhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:22.954849 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.954829 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-envoy" pod "6dabb012-d842-4c6b-8937-de5b400dbc46" (UID: "6dabb012-d842-4c6b-8937-de5b400dbc46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:22.954931 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:22.954875 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-token" pod "6dabb012-d842-4c6b-8937-de5b400dbc46" (UID: "6dabb012-d842-4c6b-8937-de5b400dbc46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:23.054130 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.054093 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:23.054130 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.054123 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjlhw\" (UniqueName: \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-kube-api-access-zjlhw\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:23.054130 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.054134 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:23.054390 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.054142 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:23.054390 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.054152 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:23.054390 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.054160 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dabb012-d842-4c6b-8937-de5b400dbc46-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:23.591072 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.591035 4304 generic.go:356] "Generic (PLEG): container finished" podID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerID="0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab" exitCode=0 Nov 03 00:43:23.591281 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.591108 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" Nov 03 00:43:23.591281 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.591123 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" event={"ID":"6dabb012-d842-4c6b-8937-de5b400dbc46","Type":"ContainerDied","Data":"0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab"} Nov 03 00:43:23.591281 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.591180 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4" event={"ID":"6dabb012-d842-4c6b-8937-de5b400dbc46","Type":"ContainerDied","Data":"74b5ea064c5c9636037a9cc07abaa3ee69216356bdbd19e3f9565884d21b8026"} Nov 03 00:43:23.591281 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.591202 4304 scope.go:122] "RemoveContainer" containerID="0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab" Nov 03 00:43:23.605433 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.605402 4304 scope.go:122] "RemoveContainer" containerID="27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9" Nov 03 00:43:23.614555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.614535 4304 scope.go:122] "RemoveContainer" containerID="0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab" Nov 03 00:43:23.614842 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:43:23.614822 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab\": container with ID starting with 0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab not found: ID does not exist" containerID="0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab" Nov 03 00:43:23.614887 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.614853 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab"} err="failed to get container status \"0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab\": rpc error: code = NotFound desc = could not find container \"0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab\": container with ID starting with 0de3d7f5de044f0988dfb2b6c266503b84b18a557c335d5e347a42faa264acab not found: ID does not exist" Nov 03 00:43:23.614887 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.614873 4304 scope.go:122] "RemoveContainer" containerID="27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9" Nov 03 00:43:23.615116 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:43:23.615096 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9\": container with ID starting with 27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9 not found: ID does not exist" containerID="27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9" Nov 03 00:43:23.615159 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.615122 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9"} err="failed to get container status \"27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9\": rpc error: code = NotFound desc = could not find container \"27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9\": container with ID starting with 27747ffdb9bf938ebd8bc46e7a13b34a1a297f564500f26774fdf7aa232810e9 not found: ID does not exist" Nov 03 00:43:23.805931 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.805903 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4"] Nov 03 00:43:23.811213 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:23.811192 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-7lff4"] Nov 03 00:43:24.093939 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.093886 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:43:24.165920 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.165891 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-envoy\") pod \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " Nov 03 00:43:24.166064 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.165959 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-token\" (UniqueName: \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-token\") pod \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " Nov 03 00:43:24.166064 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166027 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-data\") pod \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " Nov 03 00:43:24.166176 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166073 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-credential-socket\") pod \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " Nov 03 00:43:24.166176 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166148 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-kube-api-access-mnfgd\" (UniqueName: \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-kube-api-access-mnfgd\") pod \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " Nov 03 00:43:24.166287 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166184 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-socket\") pod \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " Nov 03 00:43:24.166287 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166229 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-certs\") pod \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " Nov 03 00:43:24.166287 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166261 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-podinfo\") pod \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " Nov 03 00:43:24.166439 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166342 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-credential-socket" pod "fdb726b1-3930-4f25-b566-c30c2db2d6f6" (UID: "fdb726b1-3930-4f25-b566-c30c2db2d6f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:24.166439 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166356 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istiod-ca-cert\") pod \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\" (UID: \"fdb726b1-3930-4f25-b566-c30c2db2d6f6\") " Nov 03 00:43:24.166439 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166360 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-data" pod "fdb726b1-3930-4f25-b566-c30c2db2d6f6" (UID: "fdb726b1-3930-4f25-b566-c30c2db2d6f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:24.166663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166610 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-socket" pod "fdb726b1-3930-4f25-b566-c30c2db2d6f6" (UID: "fdb726b1-3930-4f25-b566-c30c2db2d6f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:24.166753 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166641 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-certs" pod "fdb726b1-3930-4f25-b566-c30c2db2d6f6" (UID: "fdb726b1-3930-4f25-b566-c30c2db2d6f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:24.166845 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166758 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istiod-ca-cert" pod "fdb726b1-3930-4f25-b566-c30c2db2d6f6" (UID: "fdb726b1-3930-4f25-b566-c30c2db2d6f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:24.166909 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166865 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:24.166909 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166889 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:24.166909 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166903 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:24.167224 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.166918 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:24.168567 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.168536 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-podinfo" pod "fdb726b1-3930-4f25-b566-c30c2db2d6f6" (UID: "fdb726b1-3930-4f25-b566-c30c2db2d6f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:43:24.168656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.168585 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-kube-api-access-mnfgd" pod "fdb726b1-3930-4f25-b566-c30c2db2d6f6" (UID: "fdb726b1-3930-4f25-b566-c30c2db2d6f6"). InnerVolumeSpecName "kube-api-access-mnfgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:24.168754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.168739 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-envoy" pod "fdb726b1-3930-4f25-b566-c30c2db2d6f6" (UID: "fdb726b1-3930-4f25-b566-c30c2db2d6f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:24.168978 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.168953 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-token" pod "fdb726b1-3930-4f25-b566-c30c2db2d6f6" (UID: "fdb726b1-3930-4f25-b566-c30c2db2d6f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:24.268047 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.268011 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnfgd\" (UniqueName: \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-kube-api-access-mnfgd\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:24.268047 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.268042 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:24.268240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.268056 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:24.268240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.268069 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:24.268240 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.268082 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb726b1-3930-4f25-b566-c30c2db2d6f6-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:24.600082 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.600046 4304 generic.go:356] "Generic (PLEG): container finished" podID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerID="67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd" exitCode=0 Nov 03 00:43:24.600265 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.600105 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" Nov 03 00:43:24.600265 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.600136 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" event={"ID":"fdb726b1-3930-4f25-b566-c30c2db2d6f6","Type":"ContainerDied","Data":"67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd"} Nov 03 00:43:24.600265 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.600193 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h" event={"ID":"fdb726b1-3930-4f25-b566-c30c2db2d6f6","Type":"ContainerDied","Data":"df9eb66cfae057561f75ecedc4bcfa66f748fe9b11158e23d2f2971849c960fd"} Nov 03 00:43:24.600265 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.600215 4304 scope.go:122] "RemoveContainer" containerID="67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd" Nov 03 00:43:24.610132 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.610116 4304 scope.go:122] "RemoveContainer" containerID="623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d" Nov 03 00:43:24.618420 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.618405 4304 scope.go:122] "RemoveContainer" containerID="67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd" Nov 03 00:43:24.618657 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:43:24.618638 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd\": container with ID starting with 67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd not found: ID does not exist" containerID="67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd" Nov 03 00:43:24.618724 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.618671 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd"} err="failed to get container status \"67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd\": rpc error: code = NotFound desc = could not find container \"67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd\": container with ID starting with 67e4afcd332c349d8b4a0d5f5333b39093deb18c74c8779a9574b89660dcc0bd not found: ID does not exist" Nov 03 00:43:24.618724 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.618694 4304 scope.go:122] "RemoveContainer" containerID="623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d" Nov 03 00:43:24.618965 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:43:24.618946 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d\": container with ID starting with 623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d not found: ID does not exist" containerID="623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d" Nov 03 00:43:24.619013 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.618972 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d"} err="failed to get container status \"623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d\": rpc error: code = NotFound desc = could not find container \"623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d\": container with ID starting with 623bbc05715cee50e0d41ff1796dfb794a6fc006559a716ca65b01388f14132d not found: ID does not exist" Nov 03 00:43:24.763931 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.763906 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h"] Nov 03 00:43:24.769306 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.769280 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ada73-openshift-default-695b9c4b69-8jw7h"] Nov 03 00:43:24.829412 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.829379 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" path="/var/lib/kubelet/pods/6dabb012-d842-4c6b-8937-de5b400dbc46/volumes" Nov 03 00:43:24.829862 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:24.829845 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" path="/var/lib/kubelet/pods/fdb726b1-3930-4f25-b566-c30c2db2d6f6/volumes" Nov 03 00:43:28.654037 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:43:28.654001 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:43:43.637444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637369 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw"] Nov 03 00:43:43.637843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637808 4304 cpu_manager.go:414] "containerMap: removing container" podUID="e549a85b-87a0-495f-923c-3e817744f603" containerName="maas-api" Nov 03 00:43:43.637843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637821 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e549a85b-87a0-495f-923c-3e817744f603" containerName="maas-api" Nov 03 00:43:43.637843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637833 4304 cpu_manager.go:414] "containerMap: removing container" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" Nov 03 00:43:43.637843 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637838 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637848 4304 cpu_manager.go:414] "containerMap: removing container" podUID="0692bf41-be30-48ae-9a89-8e0989352a36" containerName="payload-pre-processing" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637853 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0692bf41-be30-48ae-9a89-8e0989352a36" containerName="payload-pre-processing" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637860 4304 cpu_manager.go:414] "containerMap: removing container" podUID="9a1e9db6-48e7-4868-99d4-201349b1e2b9" containerName="payload-pre-processing" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637865 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1e9db6-48e7-4868-99d4-201349b1e2b9" containerName="payload-pre-processing" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637872 4304 cpu_manager.go:414] "containerMap: removing container" podUID="bef67aae-a041-4563-866c-e803dde848fd" containerName="maas-api" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637877 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef67aae-a041-4563-866c-e803dde848fd" containerName="maas-api" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637885 4304 cpu_manager.go:414] "containerMap: removing container" podUID="5212a935-d625-48fd-a814-171f2cf68758" containerName="maas-api" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637889 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5212a935-d625-48fd-a814-171f2cf68758" containerName="maas-api" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637896 4304 cpu_manager.go:414] "containerMap: removing container" podUID="ade39df8-8a42-40ef-9652-1411ecdcb65f" containerName="payload-processing" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637900 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade39df8-8a42-40ef-9652-1411ecdcb65f" containerName="payload-processing" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637907 4304 cpu_manager.go:414] "containerMap: removing container" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637911 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637921 4304 cpu_manager.go:414] "containerMap: removing container" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637926 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637931 4304 cpu_manager.go:414] "containerMap: removing container" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637935 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637941 4304 cpu_manager.go:414] "containerMap: removing container" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637945 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637956 4304 cpu_manager.go:414] "containerMap: removing container" podUID="4902af84-c86b-46c7-a4f3-e0bac046ca77" containerName="payload-processing" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637961 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4902af84-c86b-46c7-a4f3-e0bac046ca77" containerName="payload-processing" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637969 4304 cpu_manager.go:414] "containerMap: removing container" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637973 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637982 4304 cpu_manager.go:414] "containerMap: removing container" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637987 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637993 4304 cpu_manager.go:414] "containerMap: removing container" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" Nov 03 00:43:43.637990 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.637997 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638011 4304 cpu_manager.go:414] "containerMap: removing container" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638015 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638025 4304 cpu_manager.go:414] "containerMap: removing container" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638030 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638035 4304 cpu_manager.go:414] "containerMap: removing container" podUID="01f9ade9-8108-4870-a6ae-47007de2163f" containerName="payload-pre-processing" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638040 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f9ade9-8108-4870-a6ae-47007de2163f" containerName="payload-pre-processing" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638103 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4902af84-c86b-46c7-a4f3-e0bac046ca77" containerName="payload-processing" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638110 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638116 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638121 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3765fa10-9d25-4251-b5c4-485fa784a0e1" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638128 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638134 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638140 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638144 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f9ade9-8108-4870-a6ae-47007de2163f" containerName="payload-pre-processing" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638150 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e549a85b-87a0-495f-923c-3e817744f603" containerName="maas-api" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638155 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638160 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0692bf41-be30-48ae-9a89-8e0989352a36" containerName="payload-pre-processing" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638165 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1e9db6-48e7-4868-99d4-201349b1e2b9" containerName="payload-pre-processing" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638169 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef67aae-a041-4563-866c-e803dde848fd" containerName="maas-api" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638174 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5212a935-d625-48fd-a814-171f2cf68758" containerName="maas-api" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638179 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade39df8-8a42-40ef-9652-1411ecdcb65f" containerName="payload-processing" Nov 03 00:43:43.638736 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.638184 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90a54d0-dd85-433d-a372-fecd7d8c5106" containerName="istio-proxy" Nov 03 00:43:43.641203 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.641188 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.663001 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.662976 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.663136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.663008 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09aa3322-317a-4d12-9d97-76d132c60819-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.663136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.663038 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.663235 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.663167 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z849\" (UniqueName: \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-kube-api-access-5z849\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.663235 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.663228 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.663375 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.663354 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.663437 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.663396 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.663492 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.663474 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.663599 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.663545 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09aa3322-317a-4d12-9d97-76d132c60819-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.693281 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.693255 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e28e1a-openshift-default-dockercfg-4zwkf\"" Nov 03 00:43:43.745585 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.745554 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw"] Nov 03 00:43:43.764623 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.764586 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z849\" (UniqueName: \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-kube-api-access-5z849\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.764795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.764679 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.764795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.764750 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.764930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.764797 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.764930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.764837 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.764930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.764907 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09aa3322-317a-4d12-9d97-76d132c60819-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.765094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.764937 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.765094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.764967 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09aa3322-317a-4d12-9d97-76d132c60819-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.765094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.764999 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.765261 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.765115 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.765497 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.765472 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.765672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.765499 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.765822 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.765736 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.765910 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.765859 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09aa3322-317a-4d12-9d97-76d132c60819-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.767638 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.767612 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.767819 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.767798 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09aa3322-317a-4d12-9d97-76d132c60819-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.820968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.820935 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z849\" (UniqueName: \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-kube-api-access-5z849\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.821092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.820941 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:43.953698 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:43.953673 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:44.376228 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:44.376197 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw"] Nov 03 00:43:44.377479 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:43:44.377454 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09aa3322_317a_4d12_9d97_76d132c60819.slice/crio-4c1eda88ee9a44781b26bef0c2303ac91c386284fce19e02543b17959e0b8081 WatchSource:0}: Error finding container 4c1eda88ee9a44781b26bef0c2303ac91c386284fce19e02543b17959e0b8081: Status 404 returned error can't find the container with id 4c1eda88ee9a44781b26bef0c2303ac91c386284fce19e02543b17959e0b8081 Nov 03 00:43:44.379714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:44.379681 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:43:44.379813 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:44.379746 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:43:44.379813 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:44.379794 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:43:44.714947 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:44.714907 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" event={"ID":"09aa3322-317a-4d12-9d97-76d132c60819","Type":"ContainerStarted","Data":"f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19"} Nov 03 00:43:44.715389 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:44.714965 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" event={"ID":"09aa3322-317a-4d12-9d97-76d132c60819","Type":"ContainerStarted","Data":"4c1eda88ee9a44781b26bef0c2303ac91c386284fce19e02543b17959e0b8081"} Nov 03 00:43:44.842291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:44.842218 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" podStartSLOduration=1.842196982 podStartE2EDuration="1.842196982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:44.839841571 +0000 UTC m=+2308.123399259" watchObservedRunningTime="2026-11-03 00:43:44.842196982 +0000 UTC m=+2308.125754657" Nov 03 00:43:44.953877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:44.953843 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:45.956217 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:45.956180 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:43:45.956712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:45.956248 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" podUID="09aa3322-317a-4d12-9d97-76d132c60819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 00:43:46.954868 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:46.954833 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:43:46.955059 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:46.954884 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" podUID="09aa3322-317a-4d12-9d97-76d132c60819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 00:43:47.954809 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:47.954737 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:43:47.955351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:47.954848 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" podUID="09aa3322-317a-4d12-9d97-76d132c60819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 00:43:48.953992 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:48.953961 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:43:48.954212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:48.954016 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" podUID="09aa3322-317a-4d12-9d97-76d132c60819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 00:43:49.955026 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:49.954991 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:43:49.955525 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:49.955053 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" podUID="09aa3322-317a-4d12-9d97-76d132c60819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 00:43:50.954922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:50.954884 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:43:50.955116 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:50.954965 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" podUID="09aa3322-317a-4d12-9d97-76d132c60819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 00:43:51.959639 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:51.959604 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:51.960143 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:51.960117 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:51.960759 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:43:51.960738 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:43:58.655334 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:43:58.655296 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:44:02.787193 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.787161 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l"] Nov 03 00:44:02.787617 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.787604 4304 cpu_manager.go:414] "containerMap: removing container" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" Nov 03 00:44:02.787670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.787618 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842da41d-8b5e-4197-9b92-5fa6e4d57562" containerName="istio-proxy" Nov 03 00:44:02.787724 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.787715 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="077ba44b-d69e-42f4-8511-b730c8c4855a" containerName="istio-proxy" Nov 03 00:44:02.787760 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.787726 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dabb012-d842-4c6b-8937-de5b400dbc46" containerName="istio-proxy" Nov 03 00:44:02.787760 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.787733 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb726b1-3930-4f25-b566-c30c2db2d6f6" containerName="istio-proxy" Nov 03 00:44:02.791056 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.791040 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.888536 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.888499 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l"] Nov 03 00:44:02.892327 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.892298 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.892462 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.892350 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.892524 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.892499 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202db0b8-6ff6-4493-a92f-66d68745617d-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.892586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.892569 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.892652 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.892634 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.892721 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.892705 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5xs8\" (UniqueName: \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-kube-api-access-k5xs8\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.892767 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.892748 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.892844 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.892820 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202db0b8-6ff6-4493-a92f-66d68745617d-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.892889 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.892873 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.993836 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.993805 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.993885 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202db0b8-6ff6-4493-a92f-66d68745617d-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.993934 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.993980 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994048 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.994034 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5xs8\" (UniqueName: \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-kube-api-access-k5xs8\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994273 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.994070 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994273 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.994097 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202db0b8-6ff6-4493-a92f-66d68745617d-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994273 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.994133 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994273 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.994197 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994273 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.994201 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.994322 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994533 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.994416 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994640 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.994607 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.994708 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.994686 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202db0b8-6ff6-4493-a92f-66d68745617d-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.996415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.996395 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:02.996735 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:02.996715 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202db0b8-6ff6-4493-a92f-66d68745617d-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:03.053904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:03.053828 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5xs8\" (UniqueName: \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-kube-api-access-k5xs8\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:03.056008 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:03.055981 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:03.103770 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:03.103740 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:03.530265 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:03.530240 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l"] Nov 03 00:44:03.531624 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:44:03.531600 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202db0b8_6ff6_4493_a92f_66d68745617d.slice/crio-421e877bb6c8cd76083f8312b96bf978365a0f4057d71b5de62caea384cc3858 WatchSource:0}: Error finding container 421e877bb6c8cd76083f8312b96bf978365a0f4057d71b5de62caea384cc3858: Status 404 returned error can't find the container with id 421e877bb6c8cd76083f8312b96bf978365a0f4057d71b5de62caea384cc3858 Nov 03 00:44:03.533841 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:03.533810 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:03.533951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:03.533907 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:03.534020 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:03.533954 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:03.847387 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:03.847292 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" event={"ID":"202db0b8-6ff6-4493-a92f-66d68745617d","Type":"ContainerStarted","Data":"99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a"} Nov 03 00:44:03.847894 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:03.847389 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" event={"ID":"202db0b8-6ff6-4493-a92f-66d68745617d","Type":"ContainerStarted","Data":"421e877bb6c8cd76083f8312b96bf978365a0f4057d71b5de62caea384cc3858"} Nov 03 00:44:03.977300 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:03.977247 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" podStartSLOduration=1.9772331109999999 podStartE2EDuration="1.977233111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:03.976764897 +0000 UTC m=+2327.260322572" watchObservedRunningTime="2026-11-03 00:44:03.977233111 +0000 UTC m=+2327.260790842" Nov 03 00:44:04.104603 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:04.104519 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:04.106097 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:04.106069 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 00:44:04.106176 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:04.106130 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 00:44:05.104480 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:05.104447 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 00:44:05.104928 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:05.104503 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 00:44:06.105054 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:06.105023 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 00:44:06.105470 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:06.105071 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 00:44:07.104985 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:07.104950 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 00:44:07.105168 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:07.105002 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 00:44:08.104136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:08.104104 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 00:44:08.104319 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:08.104173 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 00:44:09.104799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:09.104740 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 00:44:09.105253 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:09.104804 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 00:44:10.104045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:10.104013 4304 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 00:44:10.104261 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:10.104065 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 00:44:11.109463 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:11.109426 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:11.109868 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:11.109736 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:11.110486 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:11.110469 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:44:28.655027 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:44:28.654987 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:44:41.308086 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.308054 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn"] Nov 03 00:44:41.313564 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.313541 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.388485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.388449 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn"] Nov 03 00:44:41.419897 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.419869 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c8ce60-openshift-default-dockercfg-cbdl8\"" Nov 03 00:44:41.436177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.436141 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-envoy\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.436343 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.436227 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-token\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.436343 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.436272 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1c30b9-b6df-411c-8290-ff0a313629f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.436343 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.436320 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xzfs\" (UniqueName: \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-kube-api-access-8xzfs\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.436517 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.436380 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-data\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.436517 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.436398 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-certs\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.436517 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.436482 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-podinfo\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.436672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.436551 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-socket\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.436672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.436603 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-credential-socket\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538146 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538116 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-envoy\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538154 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-token\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538178 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1c30b9-b6df-411c-8290-ff0a313629f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538321 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538197 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xzfs\" (UniqueName: \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-kube-api-access-8xzfs\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538381 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-data\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538428 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-certs\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538458 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-podinfo\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538520 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-socket\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538586 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-credential-socket\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538762 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538715 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-data\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538929 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538859 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-certs\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538976 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538952 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-credential-socket\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.538976 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538965 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-socket\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.539094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.538990 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1c30b9-b6df-411c-8290-ff0a313629f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.540512 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.540493 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-envoy\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.541047 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.541028 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-podinfo\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.606517 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.606421 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-token\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.623012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.622393 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xzfs\" (UniqueName: \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-kube-api-access-8xzfs\") pod \"e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:41.627266 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:41.626928 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:42.062657 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:42.061734 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn"] Nov 03 00:44:42.076643 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:44:42.076617 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1c30b9_b6df_411c_8290_ff0a313629f7.slice/crio-3561b29a61cadb55f5acf5cf03d0b2c99f5f69155f783468fdecdc3e69a84914 WatchSource:0}: Error finding container 3561b29a61cadb55f5acf5cf03d0b2c99f5f69155f783468fdecdc3e69a84914: Status 404 returned error can't find the container with id 3561b29a61cadb55f5acf5cf03d0b2c99f5f69155f783468fdecdc3e69a84914 Nov 03 00:44:42.079943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:42.079901 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:42.080046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:42.079997 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:42.080099 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:42.080045 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:43.074398 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:43.074362 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" event={"ID":"4c1c30b9-b6df-411c-8290-ff0a313629f7","Type":"ContainerStarted","Data":"f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74"} Nov 03 00:44:43.074936 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:43.074411 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" event={"ID":"4c1c30b9-b6df-411c-8290-ff0a313629f7","Type":"ContainerStarted","Data":"3561b29a61cadb55f5acf5cf03d0b2c99f5f69155f783468fdecdc3e69a84914"} Nov 03 00:44:43.192255 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:43.192202 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" podStartSLOduration=2.192186993 podStartE2EDuration="2.192186993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:43.189239166 +0000 UTC m=+2366.472796880" watchObservedRunningTime="2026-11-03 00:44:43.192186993 +0000 UTC m=+2366.475744667" Nov 03 00:44:43.628650 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:43.628617 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:43.635045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:43.635014 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:44.080593 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:44.080565 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:44.081519 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:44.081501 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:44:49.764823 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.764793 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh"] Nov 03 00:44:49.768242 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.768226 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 00:44:49.825357 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.825329 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfn9n\" (UniqueName: \"kubernetes.io/projected/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-kube-api-access-lfn9n\") pod \"payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh\" (UID: \"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 00:44:49.825510 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.825383 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh\" (UID: \"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 00:44:49.891841 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.876378 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c8ce60\"" Nov 03 00:44:49.917532 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.909397 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c8ce60-dockercfg-hbpr2\"" Nov 03 00:44:49.917532 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.917163 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh"] Nov 03 00:44:49.926984 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.926949 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfn9n\" (UniqueName: \"kubernetes.io/projected/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-kube-api-access-lfn9n\") pod \"payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh\" (UID: \"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 00:44:49.927145 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.927021 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh\" (UID: \"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 00:44:49.928088 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.928056 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh\" (UID: \"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 00:44:49.937796 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.937751 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x"] Nov 03 00:44:49.941691 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.941673 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 00:44:49.973514 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.973487 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x"] Nov 03 00:44:49.984946 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:49.984917 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfn9n\" (UniqueName: \"kubernetes.io/projected/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-kube-api-access-lfn9n\") pod \"payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh\" (UID: \"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 00:44:50.028275 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:50.028201 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqh9\" (UniqueName: \"kubernetes.io/projected/10cba01f-ff75-478e-b1db-09d49dd314da-kube-api-access-2vqh9\") pod \"payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x\" (UID: \"10cba01f-ff75-478e-b1db-09d49dd314da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 00:44:50.028383 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:50.028337 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10cba01f-ff75-478e-b1db-09d49dd314da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x\" (UID: \"10cba01f-ff75-478e-b1db-09d49dd314da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 00:44:50.082036 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:50.082001 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 00:44:50.129918 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:50.129879 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqh9\" (UniqueName: \"kubernetes.io/projected/10cba01f-ff75-478e-b1db-09d49dd314da-kube-api-access-2vqh9\") pod \"payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x\" (UID: \"10cba01f-ff75-478e-b1db-09d49dd314da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 00:44:50.130100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:50.129972 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10cba01f-ff75-478e-b1db-09d49dd314da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x\" (UID: \"10cba01f-ff75-478e-b1db-09d49dd314da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 00:44:50.130691 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:50.130665 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10cba01f-ff75-478e-b1db-09d49dd314da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x\" (UID: \"10cba01f-ff75-478e-b1db-09d49dd314da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 00:44:50.186406 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:50.186378 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqh9\" (UniqueName: \"kubernetes.io/projected/10cba01f-ff75-478e-b1db-09d49dd314da-kube-api-access-2vqh9\") pod \"payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x\" (UID: \"10cba01f-ff75-478e-b1db-09d49dd314da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 00:44:50.255171 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:50.255128 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 00:44:50.456760 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:50.456730 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh"] Nov 03 00:44:50.457873 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:44:50.457838 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb0a4ee_72bb_46a1_b1f7_3038127e5ef1.slice/crio-9c3c127cf2a91aab21ea7cdd5a285cc5d9adb1a2b299ae9c3edbddcffb8c9a82 WatchSource:0}: Error finding container 9c3c127cf2a91aab21ea7cdd5a285cc5d9adb1a2b299ae9c3edbddcffb8c9a82: Status 404 returned error can't find the container with id 9c3c127cf2a91aab21ea7cdd5a285cc5d9adb1a2b299ae9c3edbddcffb8c9a82 Nov 03 00:44:50.627882 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:50.627854 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x"] Nov 03 00:44:50.629030 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:44:50.629003 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10cba01f_ff75_478e_b1db_09d49dd314da.slice/crio-2354606eb4165aaddac00fb44729b79dac22d437f018c333310997e4fa8041ec WatchSource:0}: Error finding container 2354606eb4165aaddac00fb44729b79dac22d437f018c333310997e4fa8041ec: Status 404 returned error can't find the container with id 2354606eb4165aaddac00fb44729b79dac22d437f018c333310997e4fa8041ec Nov 03 00:44:51.120817 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:51.120765 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" event={"ID":"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1","Type":"ContainerStarted","Data":"40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50"} Nov 03 00:44:51.121307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:51.120861 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 00:44:51.121307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:51.120886 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" event={"ID":"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1","Type":"ContainerStarted","Data":"9c3c127cf2a91aab21ea7cdd5a285cc5d9adb1a2b299ae9c3edbddcffb8c9a82"} Nov 03 00:44:51.122351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:51.122331 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" event={"ID":"10cba01f-ff75-478e-b1db-09d49dd314da","Type":"ContainerStarted","Data":"d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911"} Nov 03 00:44:51.122453 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:51.122369 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" event={"ID":"10cba01f-ff75-478e-b1db-09d49dd314da","Type":"ContainerStarted","Data":"2354606eb4165aaddac00fb44729b79dac22d437f018c333310997e4fa8041ec"} Nov 03 00:44:51.122515 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:51.122456 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 00:44:51.235888 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:51.235838 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" podStartSLOduration=2.23582441 podStartE2EDuration="2.23582441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:51.232625665 +0000 UTC m=+2374.516183341" watchObservedRunningTime="2026-11-03 00:44:51.23582441 +0000 UTC m=+2374.519382084" Nov 03 00:44:51.348047 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:51.347968 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" podStartSLOduration=2.347950994 podStartE2EDuration="2.347950994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:51.344528887 +0000 UTC m=+2374.628086562" watchObservedRunningTime="2026-11-03 00:44:51.347950994 +0000 UTC m=+2374.631508669" Nov 03 00:44:56.089903 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.089875 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t"] Nov 03 00:44:56.093844 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.093826 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.111125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.111102 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-token\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.111222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.111157 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-certs\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.111265 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.111230 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx94n\" (UniqueName: \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-kube-api-access-sx94n\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.111313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.111280 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-podinfo\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.111354 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.111310 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-socket\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.111475 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.111460 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff01bc5-8f42-4d85-af36-90fb2728efe4-istiod-ca-cert\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.111521 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.111511 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-data\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.111601 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.111586 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-credential-socket\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.111637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.111621 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-envoy\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.145329 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.145300 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e4c8b1-openshift-default-dockercfg-jhz42\"" Nov 03 00:44:56.199528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.199498 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t"] Nov 03 00:44:56.213038 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213006 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-data\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213201 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213049 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-credential-socket\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213201 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213073 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-envoy\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213201 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213131 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-token\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213201 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213186 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-certs\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213212 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx94n\" (UniqueName: \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-kube-api-access-sx94n\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213260 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-podinfo\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213304 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-socket\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213410 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-data\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213458 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-credential-socket\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213537 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-certs\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213656 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213418 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff01bc5-8f42-4d85-af36-90fb2728efe4-istiod-ca-cert\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.213872 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.213811 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-socket\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.214032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.214008 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff01bc5-8f42-4d85-af36-90fb2728efe4-istiod-ca-cert\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.215761 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.215738 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-envoy\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.216336 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.216315 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-podinfo\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.275190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.275099 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-token\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.277046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.277020 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx94n\" (UniqueName: \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-kube-api-access-sx94n\") pod \"e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.413550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.411036 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:56.843622 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.843598 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t"] Nov 03 00:44:56.844552 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:44:56.844528 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff01bc5_8f42_4d85_af36_90fb2728efe4.slice/crio-a3f8b841ddc401090515a04c92449e93e71115b5459fed838d3560e956e085ae WatchSource:0}: Error finding container a3f8b841ddc401090515a04c92449e93e71115b5459fed838d3560e956e085ae: Status 404 returned error can't find the container with id a3f8b841ddc401090515a04c92449e93e71115b5459fed838d3560e956e085ae Nov 03 00:44:56.846860 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.846833 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:56.846925 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.846894 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:56.846958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:56.846922 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:57.161828 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:57.161766 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" event={"ID":"dff01bc5-8f42-4d85-af36-90fb2728efe4","Type":"ContainerStarted","Data":"86d7a3b8ad60a7bd7136d55c2c93c10312db219c6d62ec891fe14a307cad1e04"} Nov 03 00:44:57.162312 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:57.161864 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" event={"ID":"dff01bc5-8f42-4d85-af36-90fb2728efe4","Type":"ContainerStarted","Data":"a3f8b841ddc401090515a04c92449e93e71115b5459fed838d3560e956e085ae"} Nov 03 00:44:57.280617 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:57.280555 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" podStartSLOduration=1.280542432 podStartE2EDuration="1.280542432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:57.277860246 +0000 UTC m=+2380.561417920" watchObservedRunningTime="2026-11-03 00:44:57.280542432 +0000 UTC m=+2380.564100106" Nov 03 00:44:57.412019 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:57.411932 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:57.418185 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:57.418140 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:58.167742 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:58.167712 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:58.168664 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:44:58.168645 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:44:58.655518 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:44:58.655475 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:45:00.205706 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.205628 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl"] Nov 03 00:45:00.209805 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.209768 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" Nov 03 00:45:00.256012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.255984 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52f3f82-029f-4fc3-8e57-a9a965ca107a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl\" (UID: \"c52f3f82-029f-4fc3-8e57-a9a965ca107a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" Nov 03 00:45:00.256223 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.256201 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbllf\" (UniqueName: \"kubernetes.io/projected/c52f3f82-029f-4fc3-8e57-a9a965ca107a-kube-api-access-nbllf\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl\" (UID: \"c52f3f82-029f-4fc3-8e57-a9a965ca107a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" Nov 03 00:45:00.288545 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.288498 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:45:00.343092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.343055 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl"] Nov 03 00:45:00.357131 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.357105 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbllf\" (UniqueName: \"kubernetes.io/projected/c52f3f82-029f-4fc3-8e57-a9a965ca107a-kube-api-access-nbllf\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl\" (UID: \"c52f3f82-029f-4fc3-8e57-a9a965ca107a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" Nov 03 00:45:00.357413 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.357388 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52f3f82-029f-4fc3-8e57-a9a965ca107a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl\" (UID: \"c52f3f82-029f-4fc3-8e57-a9a965ca107a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" Nov 03 00:45:00.358260 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.358236 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52f3f82-029f-4fc3-8e57-a9a965ca107a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl\" (UID: \"c52f3f82-029f-4fc3-8e57-a9a965ca107a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" Nov 03 00:45:00.415813 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.415770 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbllf\" (UniqueName: \"kubernetes.io/projected/c52f3f82-029f-4fc3-8e57-a9a965ca107a-kube-api-access-nbllf\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl\" (UID: \"c52f3f82-029f-4fc3-8e57-a9a965ca107a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" Nov 03 00:45:00.522988 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.522915 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" Nov 03 00:45:00.899827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:00.899800 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl"] Nov 03 00:45:00.901029 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:45:00.900996 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52f3f82_029f_4fc3_8e57_a9a965ca107a.slice/crio-46adbc6ac1e98f0fd5bfce670fee839483eeec507f5c396172fcdf8411156412 WatchSource:0}: Error finding container 46adbc6ac1e98f0fd5bfce670fee839483eeec507f5c396172fcdf8411156412: Status 404 returned error can't find the container with id 46adbc6ac1e98f0fd5bfce670fee839483eeec507f5c396172fcdf8411156412 Nov 03 00:45:01.188424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:01.188339 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" event={"ID":"c52f3f82-029f-4fc3-8e57-a9a965ca107a","Type":"ContainerStarted","Data":"46adbc6ac1e98f0fd5bfce670fee839483eeec507f5c396172fcdf8411156412"} Nov 03 00:45:02.131634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:02.131551 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 00:45:02.131634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:02.131625 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 00:45:02.197235 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:02.197200 4304 generic.go:356] "Generic (PLEG): container finished" podID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerID="49a61b0888540aac5e0237e734437ed1af96b7f2a76e17773231765912f52a6c" exitCode=60 Nov 03 00:45:02.197432 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:02.197281 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" event={"ID":"c52f3f82-029f-4fc3-8e57-a9a965ca107a","Type":"ContainerDied","Data":"49a61b0888540aac5e0237e734437ed1af96b7f2a76e17773231765912f52a6c"} Nov 03 00:45:02.197521 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:02.197497 4304 scope.go:122] "RemoveContainer" containerID="49a61b0888540aac5e0237e734437ed1af96b7f2a76e17773231765912f52a6c" Nov 03 00:45:03.205213 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:03.205170 4304 generic.go:356] "Generic (PLEG): container finished" podID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerID="01840159565e2d31944ce77f64cd497c401aa0561943ce2cf7960c43351d77e3" exitCode=60 Nov 03 00:45:03.205671 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:03.205251 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" event={"ID":"c52f3f82-029f-4fc3-8e57-a9a965ca107a","Type":"ContainerDied","Data":"01840159565e2d31944ce77f64cd497c401aa0561943ce2cf7960c43351d77e3"} Nov 03 00:45:03.205671 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:03.205343 4304 scope.go:122] "RemoveContainer" containerID="49a61b0888540aac5e0237e734437ed1af96b7f2a76e17773231765912f52a6c" Nov 03 00:45:03.205671 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:03.205602 4304 scope.go:122] "RemoveContainer" containerID="01840159565e2d31944ce77f64cd497c401aa0561943ce2cf7960c43351d77e3" Nov 03 00:45:03.205916 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:45:03.205897 4304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl_odh-ai-gateway-infra(c52f3f82-029f-4fc3-8e57-a9a965ca107a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" Nov 03 00:45:03.964674 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:03.964643 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx"] Nov 03 00:45:03.969624 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:03.969605 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 00:45:04.027640 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.027585 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e4c8b1\"" Nov 03 00:45:04.027640 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.027585 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e4c8b1-dockercfg-fq7mr\"" Nov 03 00:45:04.031528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.031505 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx"] Nov 03 00:45:04.102034 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.101955 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk9x8\" (UniqueName: \"kubernetes.io/projected/7c9c0826-a79b-44de-ab30-37a80157c40d-kube-api-access-zk9x8\") pod \"payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx\" (UID: \"7c9c0826-a79b-44de-ab30-37a80157c40d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 00:45:04.103824 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.103794 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9c0826-a79b-44de-ab30-37a80157c40d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx\" (UID: \"7c9c0826-a79b-44de-ab30-37a80157c40d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 00:45:04.205316 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.205281 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9c0826-a79b-44de-ab30-37a80157c40d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx\" (UID: \"7c9c0826-a79b-44de-ab30-37a80157c40d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 00:45:04.205761 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.205481 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk9x8\" (UniqueName: \"kubernetes.io/projected/7c9c0826-a79b-44de-ab30-37a80157c40d-kube-api-access-zk9x8\") pod \"payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx\" (UID: \"7c9c0826-a79b-44de-ab30-37a80157c40d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 00:45:04.206136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.206118 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9c0826-a79b-44de-ab30-37a80157c40d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx\" (UID: \"7c9c0826-a79b-44de-ab30-37a80157c40d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 00:45:04.212541 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.212520 4304 scope.go:122] "RemoveContainer" containerID="01840159565e2d31944ce77f64cd497c401aa0561943ce2cf7960c43351d77e3" Nov 03 00:45:04.212786 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:45:04.212751 4304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl_odh-ai-gateway-infra(c52f3f82-029f-4fc3-8e57-a9a965ca107a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" Nov 03 00:45:04.262215 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.262146 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk9x8\" (UniqueName: \"kubernetes.io/projected/7c9c0826-a79b-44de-ab30-37a80157c40d-kube-api-access-zk9x8\") pod \"payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx\" (UID: \"7c9c0826-a79b-44de-ab30-37a80157c40d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 00:45:04.283323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.283296 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 00:45:04.657170 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:04.657141 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx"] Nov 03 00:45:04.658189 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:45:04.658164 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9c0826_a79b_44de_ab30_37a80157c40d.slice/crio-260d409101c094a6da82ce6f31b0cc28890bacf189b3cd180e7f2a385de746f7 WatchSource:0}: Error finding container 260d409101c094a6da82ce6f31b0cc28890bacf189b3cd180e7f2a385de746f7: Status 404 returned error can't find the container with id 260d409101c094a6da82ce6f31b0cc28890bacf189b3cd180e7f2a385de746f7 Nov 03 00:45:05.219476 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:05.219432 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" event={"ID":"7c9c0826-a79b-44de-ab30-37a80157c40d","Type":"ContainerStarted","Data":"f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352"} Nov 03 00:45:05.219914 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:05.219493 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" event={"ID":"7c9c0826-a79b-44de-ab30-37a80157c40d","Type":"ContainerStarted","Data":"260d409101c094a6da82ce6f31b0cc28890bacf189b3cd180e7f2a385de746f7"} Nov 03 00:45:05.219914 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:05.219525 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 00:45:05.334989 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:05.334937 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" podStartSLOduration=2.334923813 podStartE2EDuration="2.334923813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:05.332497006 +0000 UTC m=+2388.616054680" watchObservedRunningTime="2026-11-03 00:45:05.334923813 +0000 UTC m=+2388.618481486" Nov 03 00:45:09.601626 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.601591 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff"] Nov 03 00:45:09.605722 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.605698 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.665478 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.665437 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-credential-socket\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.665943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.665919 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-istio-token\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.666177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.666157 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ee583e-0050-4552-92de-d56e1305fb16-istiod-ca-cert\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.666402 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.666386 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ee583e-0050-4552-92de-d56e1305fb16-istio-podinfo\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.666636 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.666617 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-data\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.666852 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.666837 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-socket\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.667052 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.667036 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6pc7\" (UniqueName: \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-kube-api-access-b6pc7\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.667269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.667252 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-certs\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.667469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.667453 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-envoy\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.718759 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.718725 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff"] Nov 03 00:45:09.718996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.718971 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-fce94b-openshift-default-dockercfg-vg67q\"" Nov 03 00:45:09.769233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769199 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-envoy\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.769424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769259 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-credential-socket\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.769424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769349 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-istio-token\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.769424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769393 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ee583e-0050-4552-92de-d56e1305fb16-istiod-ca-cert\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.769611 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769446 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ee583e-0050-4552-92de-d56e1305fb16-istio-podinfo\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.769611 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769517 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-data\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.769611 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769570 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-socket\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.769611 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769602 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6pc7\" (UniqueName: \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-kube-api-access-b6pc7\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.769864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769663 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-certs\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.769864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769712 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-credential-socket\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.769984 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769940 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-data\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.770045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.769999 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-socket\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.770102 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.770068 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-certs\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.770313 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.770276 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ee583e-0050-4552-92de-d56e1305fb16-istiod-ca-cert\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.771908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.771888 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-envoy\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.772103 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.772083 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ee583e-0050-4552-92de-d56e1305fb16-istio-podinfo\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.828490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.828451 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-istio-token\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.828490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.828487 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6pc7\" (UniqueName: \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-kube-api-access-b6pc7\") pod \"e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:09.920298 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:09.920269 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:10.359414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:10.359387 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff"] Nov 03 00:45:10.360972 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:45:10.360935 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ee583e_0050_4552_92de_d56e1305fb16.slice/crio-9465b9ccaf31796b6b0b342a0b32ca025c427e81155ad670cdb25c33b615b5e9 WatchSource:0}: Error finding container 9465b9ccaf31796b6b0b342a0b32ca025c427e81155ad670cdb25c33b615b5e9: Status 404 returned error can't find the container with id 9465b9ccaf31796b6b0b342a0b32ca025c427e81155ad670cdb25c33b615b5e9 Nov 03 00:45:10.363407 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:10.363379 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:10.363471 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:10.363442 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:10.363519 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:10.363504 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:11.262105 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:11.262068 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" event={"ID":"77ee583e-0050-4552-92de-d56e1305fb16","Type":"ContainerStarted","Data":"9479afdd7a32216474f1808663825318404c8bb950c86e3563f3e65d4c26c32d"} Nov 03 00:45:11.262516 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:11.262146 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" event={"ID":"77ee583e-0050-4552-92de-d56e1305fb16","Type":"ContainerStarted","Data":"9465b9ccaf31796b6b0b342a0b32ca025c427e81155ad670cdb25c33b615b5e9"} Nov 03 00:45:11.378561 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:11.378497 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" podStartSLOduration=2.378479814 podStartE2EDuration="2.378479814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:11.377646816 +0000 UTC m=+2394.661204490" watchObservedRunningTime="2026-11-03 00:45:11.378479814 +0000 UTC m=+2394.662037490" Nov 03 00:45:11.921205 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:11.921176 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:11.927015 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:11.926993 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:12.267841 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:12.267748 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:12.268622 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:12.268603 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:45:14.822715 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:14.822685 4304 scope.go:122] "RemoveContainer" containerID="01840159565e2d31944ce77f64cd497c401aa0561943ce2cf7960c43351d77e3" Nov 03 00:45:15.287174 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:15.287136 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" event={"ID":"c52f3f82-029f-4fc3-8e57-a9a965ca107a","Type":"ContainerStarted","Data":"1f2a4b49a155b52857aea100c43b9812b2534ace688b8a284977bb9c61e444e3"} Nov 03 00:45:15.401701 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:15.401651 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" podStartSLOduration=15.401636815 podStartE2EDuration="15.401636815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:15.399310375 +0000 UTC m=+2398.682868049" watchObservedRunningTime="2026-11-03 00:45:15.401636815 +0000 UTC m=+2398.685194519" Nov 03 00:45:16.228373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:16.228342 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 00:45:16.295696 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:16.295666 4304 generic.go:356] "Generic (PLEG): container finished" podID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerID="1f2a4b49a155b52857aea100c43b9812b2534ace688b8a284977bb9c61e444e3" exitCode=60 Nov 03 00:45:16.295879 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:16.295728 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" event={"ID":"c52f3f82-029f-4fc3-8e57-a9a965ca107a","Type":"ContainerDied","Data":"1f2a4b49a155b52857aea100c43b9812b2534ace688b8a284977bb9c61e444e3"} Nov 03 00:45:16.295879 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:16.295789 4304 scope.go:122] "RemoveContainer" containerID="01840159565e2d31944ce77f64cd497c401aa0561943ce2cf7960c43351d77e3" Nov 03 00:45:16.296122 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:16.296101 4304 scope.go:122] "RemoveContainer" containerID="1f2a4b49a155b52857aea100c43b9812b2534ace688b8a284977bb9c61e444e3" Nov 03 00:45:16.296408 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:45:16.296382 4304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl_odh-ai-gateway-infra(c52f3f82-029f-4fc3-8e57-a9a965ca107a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" Nov 03 00:45:16.412707 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:16.412673 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl"] Nov 03 00:45:17.498621 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:17.498597 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" Nov 03 00:45:17.661754 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:17.661721 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52f3f82-029f-4fc3-8e57-a9a965ca107a-kube-api-access-nbllf\" (UniqueName: \"kubernetes.io/projected/c52f3f82-029f-4fc3-8e57-a9a965ca107a-kube-api-access-nbllf\") pod \"c52f3f82-029f-4fc3-8e57-a9a965ca107a\" (UID: \"c52f3f82-029f-4fc3-8e57-a9a965ca107a\") " Nov 03 00:45:17.662125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:17.662108 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52f3f82-029f-4fc3-8e57-a9a965ca107a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52f3f82-029f-4fc3-8e57-a9a965ca107a-maas-api-service-ca\") pod \"c52f3f82-029f-4fc3-8e57-a9a965ca107a\" (UID: \"c52f3f82-029f-4fc3-8e57-a9a965ca107a\") " Nov 03 00:45:17.662833 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:17.662513 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52f3f82-029f-4fc3-8e57-a9a965ca107a-maas-api-service-ca" pod "c52f3f82-029f-4fc3-8e57-a9a965ca107a" (UID: "c52f3f82-029f-4fc3-8e57-a9a965ca107a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:17.663135 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:17.663118 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52f3f82-029f-4fc3-8e57-a9a965ca107a-maas-api-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:17.667421 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:17.667372 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52f3f82-029f-4fc3-8e57-a9a965ca107a-kube-api-access-nbllf" pod "c52f3f82-029f-4fc3-8e57-a9a965ca107a" (UID: "c52f3f82-029f-4fc3-8e57-a9a965ca107a"). InnerVolumeSpecName "kube-api-access-nbllf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:17.764536 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:17.764505 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbllf\" (UniqueName: \"kubernetes.io/projected/c52f3f82-029f-4fc3-8e57-a9a965ca107a-kube-api-access-nbllf\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:18.311968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:18.311935 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" Nov 03 00:45:18.311968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:18.311944 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl" event={"ID":"c52f3f82-029f-4fc3-8e57-a9a965ca107a","Type":"ContainerDied","Data":"46adbc6ac1e98f0fd5bfce670fee839483eeec507f5c396172fcdf8411156412"} Nov 03 00:45:18.312208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:18.312008 4304 scope.go:122] "RemoveContainer" containerID="1f2a4b49a155b52857aea100c43b9812b2534ace688b8a284977bb9c61e444e3" Nov 03 00:45:18.476042 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:18.476006 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl"] Nov 03 00:45:18.525238 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:18.525207 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894445-d9gwl"] Nov 03 00:45:18.829307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:18.829272 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" path="/var/lib/kubelet/pods/c52f3f82-029f-4fc3-8e57-a9a965ca107a/volumes" Nov 03 00:45:21.556944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.556904 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n"] Nov 03 00:45:21.557745 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.557720 4304 cpu_manager.go:414] "containerMap: removing container" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerName="cleanup" Nov 03 00:45:21.557904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.557748 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerName="cleanup" Nov 03 00:45:21.557904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.557765 4304 cpu_manager.go:414] "containerMap: removing container" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerName="cleanup" Nov 03 00:45:21.557904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.557794 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerName="cleanup" Nov 03 00:45:21.557904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.557809 4304 cpu_manager.go:414] "containerMap: removing container" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerName="cleanup" Nov 03 00:45:21.557904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.557817 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerName="cleanup" Nov 03 00:45:21.558153 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.557986 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerName="cleanup" Nov 03 00:45:21.558153 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.558001 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerName="cleanup" Nov 03 00:45:21.558153 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.558011 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52f3f82-029f-4fc3-8e57-a9a965ca107a" containerName="cleanup" Nov 03 00:45:21.563598 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.563578 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:21.616838 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.616801 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n"] Nov 03 00:45:21.662939 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.662906 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-b2e795-serving-cert\"" Nov 03 00:45:21.707410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.707363 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c8feee-4ebf-4910-9851-0744e618c916-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:21.707595 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.707542 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:21.707595 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.707578 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:21.707728 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.707689 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vscjk\" (UniqueName: \"kubernetes.io/projected/63c8feee-4ebf-4910-9851-0744e618c916-kube-api-access-vscjk\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:21.808935 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.808847 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c8feee-4ebf-4910-9851-0744e618c916-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:21.809100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.808972 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:21.809100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.809004 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:21.809100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.809059 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vscjk\" (UniqueName: \"kubernetes.io/projected/63c8feee-4ebf-4910-9851-0744e618c916-kube-api-access-vscjk\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:21.809272 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:45:21.809179 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-serving-cert: secret "maas-api-e2e-shared-a-w1-b2e795-serving-cert" not found Nov 03 00:45:21.809272 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:45:21.809245 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-maas-api-tls podName:63c8feee-4ebf-4910-9851-0744e618c916 nodeName:}" failed. No retries permitted until 2026-11-03 00:45:22.309226594 +0000 UTC m=+2405.592784246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-maas-api-tls") pod "maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" (UID: "63c8feee-4ebf-4910-9851-0744e618c916") : secret "maas-api-e2e-shared-a-w1-b2e795-serving-cert" not found Nov 03 00:45:21.809517 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.809497 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c8feee-4ebf-4910-9851-0744e618c916-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:21.811806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.811763 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:21.866566 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:21.866488 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vscjk\" (UniqueName: \"kubernetes.io/projected/63c8feee-4ebf-4910-9851-0744e618c916-kube-api-access-vscjk\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:22.313966 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:22.313932 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:22.316506 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:22.316473 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:22.480999 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:22.480961 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:22.854890 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:22.854857 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n"] Nov 03 00:45:22.855320 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:45:22.855181 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c8feee_4ebf_4910_9851_0744e618c916.slice/crio-70a642f68c42610cccdfa9b86b4a9b4b0262dfd8023dca882a81edb042a77fbf WatchSource:0}: Error finding container 70a642f68c42610cccdfa9b86b4a9b4b0262dfd8023dca882a81edb042a77fbf: Status 404 returned error can't find the container with id 70a642f68c42610cccdfa9b86b4a9b4b0262dfd8023dca882a81edb042a77fbf Nov 03 00:45:22.856681 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:22.856663 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:45:23.348290 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:23.348252 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" event={"ID":"63c8feee-4ebf-4910-9851-0744e618c916","Type":"ContainerStarted","Data":"70a642f68c42610cccdfa9b86b4a9b4b0262dfd8023dca882a81edb042a77fbf"} Nov 03 00:45:24.355852 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:24.355815 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" event={"ID":"63c8feee-4ebf-4910-9851-0744e618c916","Type":"ContainerStarted","Data":"647a0599096016f7ddf09971b7e2b83f04536af8cfc62e3e569873cb7ac97d8c"} Nov 03 00:45:24.356270 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:24.355954 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:24.471666 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:24.471608 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" podStartSLOduration=2.614450315 podStartE2EDuration="3.471594039s" totalImagesPullingTime="857.143724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:24.468805238 +0000 UTC m=+2407.752362904" watchObservedRunningTime="2026-11-03 00:45:24.471594039 +0000 UTC m=+2407.755151714" Nov 03 00:45:28.654296 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:45:28.654263 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:45:30.367733 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:30.367698 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 00:45:31.159456 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.159425 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp"] Nov 03 00:45:31.163044 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.163026 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.212444 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.212414 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwmrc\" (UniqueName: \"kubernetes.io/projected/321e724f-037e-4159-ab48-4b852a01f18f-kube-api-access-zwmrc\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.212577 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.212463 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.212648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.212631 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321e724f-037e-4159-ab48-4b852a01f18f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.212763 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.212746 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.219817 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.219798 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c8ce60-serving-cert\"" Nov 03 00:45:31.271394 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.271369 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp"] Nov 03 00:45:31.314052 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.314024 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwmrc\" (UniqueName: \"kubernetes.io/projected/321e724f-037e-4159-ab48-4b852a01f18f-kube-api-access-zwmrc\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.314212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.314069 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.314212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.314136 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321e724f-037e-4159-ab48-4b852a01f18f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.314328 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.314226 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.314380 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:45:31.314343 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-serving-cert: secret "maas-api-e2e-shared-a-w2-c8ce60-serving-cert" not found Nov 03 00:45:31.314423 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:45:31.314412 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-maas-api-tls podName:321e724f-037e-4159-ab48-4b852a01f18f nodeName:}" failed. No retries permitted until 2026-11-03 00:45:31.814390877 +0000 UTC m=+2415.097948533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-maas-api-tls") pod "maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" (UID: "321e724f-037e-4159-ab48-4b852a01f18f") : secret "maas-api-e2e-shared-a-w2-c8ce60-serving-cert" not found Nov 03 00:45:31.315001 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.314907 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321e724f-037e-4159-ab48-4b852a01f18f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.316636 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.316616 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.377876 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.377848 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwmrc\" (UniqueName: \"kubernetes.io/projected/321e724f-037e-4159-ab48-4b852a01f18f-kube-api-access-zwmrc\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.819899 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.819865 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:31.822411 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:31.822392 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:32.076092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:32.076018 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:32.448472 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:32.448440 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp"] Nov 03 00:45:32.449476 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:45:32.449448 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321e724f_037e_4159_ab48_4b852a01f18f.slice/crio-e722d3addc911ff72e890c6909f63299b28771e46f557b7a83878876e42bbcc5 WatchSource:0}: Error finding container e722d3addc911ff72e890c6909f63299b28771e46f557b7a83878876e42bbcc5: Status 404 returned error can't find the container with id e722d3addc911ff72e890c6909f63299b28771e46f557b7a83878876e42bbcc5 Nov 03 00:45:33.413943 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:33.413906 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" event={"ID":"321e724f-037e-4159-ab48-4b852a01f18f","Type":"ContainerStarted","Data":"b003938afe9cef5cfa8fbbe80f1259dc43c9d3379af3c869681036101ccd6716"} Nov 03 00:45:33.414120 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:33.413974 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:33.414120 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:33.413996 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" event={"ID":"321e724f-037e-4159-ab48-4b852a01f18f","Type":"ContainerStarted","Data":"e722d3addc911ff72e890c6909f63299b28771e46f557b7a83878876e42bbcc5"} Nov 03 00:45:33.555827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:33.555670 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" podStartSLOduration=1.689129473 podStartE2EDuration="2.555656145s" totalImagesPullingTime="866.526672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:33.554389673 +0000 UTC m=+2416.837947371" watchObservedRunningTime="2026-11-03 00:45:33.555656145 +0000 UTC m=+2416.839213820" Nov 03 00:45:39.426740 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:39.426700 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 00:45:43.658687 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.658650 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz"] Nov 03 00:45:43.663795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.663760 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.763938 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.763900 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.764136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.764020 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.764136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.764083 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.764136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.764117 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076018d2-c750-4a75-ad71-acf5d701136a-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.764308 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.764188 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.764308 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.764225 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.764308 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.764259 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.764308 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.764284 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlw28\" (UniqueName: \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-kube-api-access-tlw28\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.764521 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.764327 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz"] Nov 03 00:45:43.764521 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.764355 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076018d2-c750-4a75-ad71-acf5d701136a-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.865425 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.865391 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.865425 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.865427 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.865668 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.865442 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlw28\" (UniqueName: \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-kube-api-access-tlw28\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.865668 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.865536 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076018d2-c750-4a75-ad71-acf5d701136a-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.865668 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.865608 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.865885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.865691 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.865885 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.865751 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.865981 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.865883 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076018d2-c750-4a75-ad71-acf5d701136a-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.865981 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.865948 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.866197 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.866164 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.866197 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.866178 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.866304 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.866273 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.866643 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.866626 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.866869 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.866843 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076018d2-c750-4a75-ad71-acf5d701136a-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.868052 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.868034 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.868334 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.868315 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076018d2-c750-4a75-ad71-acf5d701136a-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.923470 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.923413 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.923710 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.923686 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlw28\" (UniqueName: \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-kube-api-access-tlw28\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:43.976487 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:43.976465 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:44.399575 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:45:44.399547 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076018d2_c750_4a75_ad71_acf5d701136a.slice/crio-a817cb77831cd86da235648efde0aa44a1f11b3f9c1ac66479794fb5404129c6 WatchSource:0}: Error finding container a817cb77831cd86da235648efde0aa44a1f11b3f9c1ac66479794fb5404129c6: Status 404 returned error can't find the container with id a817cb77831cd86da235648efde0aa44a1f11b3f9c1ac66479794fb5404129c6 Nov 03 00:45:44.401429 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:44.401408 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz"] Nov 03 00:45:44.401911 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:44.401881 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:44.401959 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:44.401946 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:44.402013 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:44.401978 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:44.483225 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:44.483183 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" event={"ID":"076018d2-c750-4a75-ad71-acf5d701136a","Type":"ContainerStarted","Data":"f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce"} Nov 03 00:45:44.483401 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:44.483260 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" event={"ID":"076018d2-c750-4a75-ad71-acf5d701136a","Type":"ContainerStarted","Data":"a817cb77831cd86da235648efde0aa44a1f11b3f9c1ac66479794fb5404129c6"} Nov 03 00:45:44.599973 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:44.599908 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" podStartSLOduration=1.5998897680000002 podStartE2EDuration="1.599889768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:44.597229688 +0000 UTC m=+2427.880787359" watchObservedRunningTime="2026-11-03 00:45:44.599889768 +0000 UTC m=+2427.883447444" Nov 03 00:45:44.977638 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:44.977594 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:45.978438 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:45.978402 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:45:45.978917 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:45.978456 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" podUID="076018d2-c750-4a75-ad71-acf5d701136a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 00:45:46.977798 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:46.977050 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:45:46.977798 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:46.977111 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" podUID="076018d2-c750-4a75-ad71-acf5d701136a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 00:45:47.977607 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:47.977572 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:45:47.980032 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:47.977622 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" podUID="076018d2-c750-4a75-ad71-acf5d701136a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 00:45:48.979010 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:48.978965 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:45:48.979950 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:48.979920 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" podUID="076018d2-c750-4a75-ad71-acf5d701136a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 00:45:49.977548 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:49.977511 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:45:49.977758 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:49.977572 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" podUID="076018d2-c750-4a75-ad71-acf5d701136a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 00:45:50.977682 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:50.977637 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:45:50.978237 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:50.977689 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" podUID="076018d2-c750-4a75-ad71-acf5d701136a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 00:45:52.259909 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:52.259872 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:52.260386 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:52.260349 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:52.261317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:52.261295 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:45:58.638236 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:58.638198 4304 generic.go:356] "Generic (PLEG): container finished" podID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerID="f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74" exitCode=0 Nov 03 00:45:58.638820 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:58.638268 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" event={"ID":"4c1c30b9-b6df-411c-8290-ff0a313629f7","Type":"ContainerDied","Data":"f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74"} Nov 03 00:45:58.638820 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:58.638591 4304 scope.go:122] "RemoveContainer" containerID="f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74" Nov 03 00:45:58.640140 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:58.640113 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:58.640241 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:58.640196 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:58.640279 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:58.640242 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:58.656109 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:45:58.656084 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:45:59.080969 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:59.080934 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:45:59.653303 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:59.653254 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" event={"ID":"4c1c30b9-b6df-411c-8290-ff0a313629f7","Type":"ContainerStarted","Data":"a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034"} Nov 03 00:45:59.817164 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:59.817111 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx"] Nov 03 00:45:59.817412 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:45:59.817355 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" podUID="77751960-fc94-4513-a20c-c76a475834f0" containerName="maas-api" containerID="cri-o://632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95" gracePeriod=30 Nov 03 00:46:00.157960 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.157937 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:46:00.293179 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.293096 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-maas-api-tls\") pod \"77751960-fc94-4513-a20c-c76a475834f0\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " Nov 03 00:46:00.293373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.293256 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77751960-fc94-4513-a20c-c76a475834f0-kube-api-access-vg8wl\" (UniqueName: \"kubernetes.io/projected/77751960-fc94-4513-a20c-c76a475834f0-kube-api-access-vg8wl\") pod \"77751960-fc94-4513-a20c-c76a475834f0\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " Nov 03 00:46:00.293373 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.293367 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-metrics-certs\") pod \"77751960-fc94-4513-a20c-c76a475834f0\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " Nov 03 00:46:00.293507 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.293441 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77751960-fc94-4513-a20c-c76a475834f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77751960-fc94-4513-a20c-c76a475834f0-openshift-service-ca\") pod \"77751960-fc94-4513-a20c-c76a475834f0\" (UID: \"77751960-fc94-4513-a20c-c76a475834f0\") " Nov 03 00:46:00.294101 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.294015 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77751960-fc94-4513-a20c-c76a475834f0-openshift-service-ca" pod "77751960-fc94-4513-a20c-c76a475834f0" (UID: "77751960-fc94-4513-a20c-c76a475834f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:00.295575 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.295553 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77751960-fc94-4513-a20c-c76a475834f0-kube-api-access-vg8wl" pod "77751960-fc94-4513-a20c-c76a475834f0" (UID: "77751960-fc94-4513-a20c-c76a475834f0"). InnerVolumeSpecName "kube-api-access-vg8wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:00.295940 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.295920 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-maas-api-tls" pod "77751960-fc94-4513-a20c-c76a475834f0" (UID: "77751960-fc94-4513-a20c-c76a475834f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:00.296057 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.296042 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-metrics-certs" pod "77751960-fc94-4513-a20c-c76a475834f0" (UID: "77751960-fc94-4513-a20c-c76a475834f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:00.395531 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.395495 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg8wl\" (UniqueName: \"kubernetes.io/projected/77751960-fc94-4513-a20c-c76a475834f0-kube-api-access-vg8wl\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:00.395531 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.395529 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:00.395531 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.395539 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77751960-fc94-4513-a20c-c76a475834f0-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:00.395865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.395547 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77751960-fc94-4513-a20c-c76a475834f0-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:00.631857 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.629002 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:46:00.631857 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.629152 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 00:46:00.631857 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.629196 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 00:46:00.661621 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.661583 4304 generic.go:356] "Generic (PLEG): container finished" podID="77751960-fc94-4513-a20c-c76a475834f0" containerID="632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95" exitCode=0 Nov 03 00:46:00.662128 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.661642 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" Nov 03 00:46:00.662128 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.661666 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" event={"ID":"77751960-fc94-4513-a20c-c76a475834f0","Type":"ContainerDied","Data":"632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95"} Nov 03 00:46:00.662128 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.661723 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx" event={"ID":"77751960-fc94-4513-a20c-c76a475834f0","Type":"ContainerDied","Data":"02ba1ac19c97278354b3b2eb5660a4d70823a5b53e9b07f7ac9106924a248972"} Nov 03 00:46:00.662128 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.661746 4304 scope.go:122] "RemoveContainer" containerID="632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95" Nov 03 00:46:00.671968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.671952 4304 scope.go:122] "RemoveContainer" containerID="632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95" Nov 03 00:46:00.672226 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:46:00.672206 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95\": container with ID starting with 632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95 not found: ID does not exist" containerID="632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95" Nov 03 00:46:00.672266 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.672235 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95"} err="failed to get container status \"632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95\": rpc error: code = NotFound desc = could not find container \"632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95\": container with ID starting with 632a0a77164accfe37c4a997256b6da9b517558d4c7b6d540c34f3c783fccc95 not found: ID does not exist" Nov 03 00:46:00.856637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.856609 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx"] Nov 03 00:46:00.887250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:00.887178 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d61eae-867f8c75d-nshxx"] Nov 03 00:46:01.637806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:01.635234 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 00:46:01.637806 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:01.635293 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 00:46:02.637799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:02.634290 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 00:46:02.637799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:02.634381 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 00:46:02.851443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:02.851404 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77751960-fc94-4513-a20c-c76a475834f0" path="/var/lib/kubelet/pods/77751960-fc94-4513-a20c-c76a475834f0/volumes" Nov 03 00:46:03.635237 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:03.635199 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 00:46:03.635559 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:03.635529 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 00:46:04.628338 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:04.628300 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 00:46:04.628725 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:04.628370 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 00:46:05.633109 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:05.633080 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:46:05.633476 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:05.633449 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:46:05.634091 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:05.634071 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 00:46:06.542528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.542494 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s"] Nov 03 00:46:06.543033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.543019 4304 cpu_manager.go:414] "containerMap: removing container" podUID="77751960-fc94-4513-a20c-c76a475834f0" containerName="maas-api" Nov 03 00:46:06.543081 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.543034 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77751960-fc94-4513-a20c-c76a475834f0" containerName="maas-api" Nov 03 00:46:06.543125 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.543117 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="77751960-fc94-4513-a20c-c76a475834f0" containerName="maas-api" Nov 03 00:46:06.547710 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.547690 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:06.596667 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.596639 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gfrt\" (UniqueName: \"kubernetes.io/projected/8896996a-4482-4880-a631-3ab45648906b-kube-api-access-8gfrt\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:06.596835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.596704 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:06.596936 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.596885 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:06.596996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.596945 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8896996a-4482-4880-a631-3ab45648906b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:06.601755 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.601727 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-fce94b-serving-cert\"" Nov 03 00:46:06.608713 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.608692 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s"] Nov 03 00:46:06.697674 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.697635 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:06.698149 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.697695 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:06.698149 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.697732 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8896996a-4482-4880-a631-3ab45648906b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:06.698149 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:46:06.697875 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-serving-cert: secret "maas-api-e2e-shared-a-w4-fce94b-serving-cert" not found Nov 03 00:46:06.698149 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.697879 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gfrt\" (UniqueName: \"kubernetes.io/projected/8896996a-4482-4880-a631-3ab45648906b-kube-api-access-8gfrt\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:06.698149 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:46:06.697950 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-maas-api-tls podName:8896996a-4482-4880-a631-3ab45648906b nodeName:}" failed. No retries permitted until 2026-11-03 00:46:07.197925016 +0000 UTC m=+2450.481482668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-maas-api-tls") pod "maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" (UID: "8896996a-4482-4880-a631-3ab45648906b") : secret "maas-api-e2e-shared-a-w4-fce94b-serving-cert" not found Nov 03 00:46:06.698548 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.698527 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8896996a-4482-4880-a631-3ab45648906b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:06.700146 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.700129 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:06.755233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:06.755210 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gfrt\" (UniqueName: \"kubernetes.io/projected/8896996a-4482-4880-a631-3ab45648906b-kube-api-access-8gfrt\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:07.207232 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:07.207203 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:07.209737 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:07.209714 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:07.460411 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:07.460321 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:07.837011 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:07.836984 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s"] Nov 03 00:46:07.837978 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:46:07.837953 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8896996a_4482_4880_a631_3ab45648906b.slice/crio-17414323b905963c2fb94d1c2a35868e980ae967373a90b52a66e6141f8570d9 WatchSource:0}: Error finding container 17414323b905963c2fb94d1c2a35868e980ae967373a90b52a66e6141f8570d9: Status 404 returned error can't find the container with id 17414323b905963c2fb94d1c2a35868e980ae967373a90b52a66e6141f8570d9 Nov 03 00:46:08.719835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:08.719803 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" event={"ID":"8896996a-4482-4880-a631-3ab45648906b","Type":"ContainerStarted","Data":"17414323b905963c2fb94d1c2a35868e980ae967373a90b52a66e6141f8570d9"} Nov 03 00:46:09.727936 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:09.727901 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" event={"ID":"8896996a-4482-4880-a631-3ab45648906b","Type":"ContainerStarted","Data":"6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f"} Nov 03 00:46:09.728369 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:09.728145 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:09.843591 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:09.843541 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" podStartSLOduration=3.02849243 podStartE2EDuration="3.843527811s" totalImagesPullingTime="815.035381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:09.840620454 +0000 UTC m=+2453.124178128" watchObservedRunningTime="2026-11-03 00:46:09.843527811 +0000 UTC m=+2453.127085482" Nov 03 00:46:11.971834 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:11.971800 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m"] Nov 03 00:46:15.740288 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:15.740260 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 00:46:16.972315 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:16.972274 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" podUID="c2331374-06e2-49fe-8f5f-5501302327ab" containerName="payload-processing" containerID="cri-o://71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e" gracePeriod=25 Nov 03 00:46:17.283494 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.283469 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:46:17.463279 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.463249 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2331374-06e2-49fe-8f5f-5501302327ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2331374-06e2-49fe-8f5f-5501302327ab-plugins-config-volume\") pod \"c2331374-06e2-49fe-8f5f-5501302327ab\" (UID: \"c2331374-06e2-49fe-8f5f-5501302327ab\") " Nov 03 00:46:17.463467 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.463290 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2331374-06e2-49fe-8f5f-5501302327ab-kube-api-access-dr6k2\" (UniqueName: \"kubernetes.io/projected/c2331374-06e2-49fe-8f5f-5501302327ab-kube-api-access-dr6k2\") pod \"c2331374-06e2-49fe-8f5f-5501302327ab\" (UID: \"c2331374-06e2-49fe-8f5f-5501302327ab\") " Nov 03 00:46:17.463689 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.463662 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2331374-06e2-49fe-8f5f-5501302327ab-plugins-config-volume" pod "c2331374-06e2-49fe-8f5f-5501302327ab" (UID: "c2331374-06e2-49fe-8f5f-5501302327ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:17.466275 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.466252 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2331374-06e2-49fe-8f5f-5501302327ab-kube-api-access-dr6k2" pod "c2331374-06e2-49fe-8f5f-5501302327ab" (UID: "c2331374-06e2-49fe-8f5f-5501302327ab"). InnerVolumeSpecName "kube-api-access-dr6k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:17.565291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.565205 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2331374-06e2-49fe-8f5f-5501302327ab-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:17.565291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.565238 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr6k2\" (UniqueName: \"kubernetes.io/projected/c2331374-06e2-49fe-8f5f-5501302327ab-kube-api-access-dr6k2\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:17.775733 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.775701 4304 generic.go:356] "Generic (PLEG): container finished" podID="c2331374-06e2-49fe-8f5f-5501302327ab" containerID="71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e" exitCode=0 Nov 03 00:46:17.775733 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.775748 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" Nov 03 00:46:17.775999 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.775801 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" event={"ID":"c2331374-06e2-49fe-8f5f-5501302327ab","Type":"ContainerDied","Data":"71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e"} Nov 03 00:46:17.775999 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.775858 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m" event={"ID":"c2331374-06e2-49fe-8f5f-5501302327ab","Type":"ContainerDied","Data":"d025ed0a4dd476933e64538b2c2be9bc81b42e5b8893e5c9ddcf82bb7bf6ddb3"} Nov 03 00:46:17.775999 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.775879 4304 scope.go:122] "RemoveContainer" containerID="71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e" Nov 03 00:46:17.786064 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.786046 4304 scope.go:122] "RemoveContainer" containerID="71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e" Nov 03 00:46:17.786338 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:46:17.786319 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e\": container with ID starting with 71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e not found: ID does not exist" containerID="71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e" Nov 03 00:46:17.786392 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.786346 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e"} err="failed to get container status \"71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e\": rpc error: code = NotFound desc = could not find container \"71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e\": container with ID starting with 71a560854cc445e24e2eff7f4175417ae876bc771c95993ff6ef23cb3169449e not found: ID does not exist" Nov 03 00:46:17.954504 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.954466 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m"] Nov 03 00:46:17.958220 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:17.958197 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bdd194-6b9868b54f-vrn6m"] Nov 03 00:46:18.852074 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:18.852040 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2331374-06e2-49fe-8f5f-5501302327ab" path="/var/lib/kubelet/pods/c2331374-06e2-49fe-8f5f-5501302327ab/volumes" Nov 03 00:46:24.820170 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:24.820134 4304 generic.go:356] "Generic (PLEG): container finished" podID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerID="86d7a3b8ad60a7bd7136d55c2c93c10312db219c6d62ec891fe14a307cad1e04" exitCode=0 Nov 03 00:46:24.820580 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:24.820203 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" event={"ID":"dff01bc5-8f42-4d85-af36-90fb2728efe4","Type":"ContainerDied","Data":"86d7a3b8ad60a7bd7136d55c2c93c10312db219c6d62ec891fe14a307cad1e04"} Nov 03 00:46:24.820646 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:24.820630 4304 scope.go:122] "RemoveContainer" containerID="86d7a3b8ad60a7bd7136d55c2c93c10312db219c6d62ec891fe14a307cad1e04" Nov 03 00:46:24.822290 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:24.822260 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:24.822380 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:24.822348 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:24.822417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:24.822396 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:25.851575 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:25.851536 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" event={"ID":"dff01bc5-8f42-4d85-af36-90fb2728efe4","Type":"ContainerStarted","Data":"42e2019fee0da551093eb3a596e7dd130e82b3ad6c6f110480bc0e67dd531cba"} Nov 03 00:46:26.411930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:26.411893 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:46:26.413381 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:26.413355 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:46:26.413507 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:26.413409 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 00:46:27.412402 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:27.412360 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:46:27.412854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:27.412417 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 00:46:28.168555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:28.168521 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:46:28.411750 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:28.411717 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:46:28.411941 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:28.411790 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 00:46:28.670929 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:46:28.670893 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:46:29.411717 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:29.411641 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:46:29.411930 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:29.411817 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 00:46:30.411828 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:30.411770 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:46:30.412243 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:30.411876 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 00:46:31.421238 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:31.421202 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:46:31.422667 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:31.422648 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 00:46:48.023936 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:48.023902 4304 generic.go:356] "Generic (PLEG): container finished" podID="77ee583e-0050-4552-92de-d56e1305fb16" containerID="9479afdd7a32216474f1808663825318404c8bb950c86e3563f3e65d4c26c32d" exitCode=0 Nov 03 00:46:48.024678 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:48.023981 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" event={"ID":"77ee583e-0050-4552-92de-d56e1305fb16","Type":"ContainerDied","Data":"9479afdd7a32216474f1808663825318404c8bb950c86e3563f3e65d4c26c32d"} Nov 03 00:46:48.024678 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:48.024338 4304 scope.go:122] "RemoveContainer" containerID="9479afdd7a32216474f1808663825318404c8bb950c86e3563f3e65d4c26c32d" Nov 03 00:46:48.025732 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:48.025703 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:48.025809 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:48.025798 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:48.025858 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:48.025835 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:49.032489 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:49.032449 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" event={"ID":"77ee583e-0050-4552-92de-d56e1305fb16","Type":"ContainerStarted","Data":"4f0ea5c9633dece32dd8b8916509f32ec5df53d6c3994a4613c73aeaf3b4f80a"} Nov 03 00:46:49.924103 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:49.924071 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:46:49.927865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:49.924366 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:46:49.934907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:49.934873 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 00:46:50.932153 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:50.932119 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:46:50.932728 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:50.932697 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 00:46:51.921269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:51.920768 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:46:51.921269 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:51.920893 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 00:46:52.941800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:52.941054 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:46:52.941800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:52.941122 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 00:46:53.920599 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:53.920564 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:46:53.920803 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:53.920631 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 00:46:54.924638 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:54.924611 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:46:54.925083 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:54.924980 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:46:54.925666 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:46:54.925647 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 00:46:58.712670 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:46:58.712633 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:47:21.692484 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.692449 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq"] Nov 03 00:47:21.693039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.693023 4304 cpu_manager.go:414] "containerMap: removing container" podUID="c2331374-06e2-49fe-8f5f-5501302327ab" containerName="payload-processing" Nov 03 00:47:21.693089 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.693041 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2331374-06e2-49fe-8f5f-5501302327ab" containerName="payload-processing" Nov 03 00:47:21.693157 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.693148 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2331374-06e2-49fe-8f5f-5501302327ab" containerName="payload-processing" Nov 03 00:47:21.696387 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.696368 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 00:47:21.748664 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.748632 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-20f0a3\"" Nov 03 00:47:21.799084 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.799023 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-20f0a3-dockercfg-p2d9k\"" Nov 03 00:47:21.805105 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.805071 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq"] Nov 03 00:47:21.854796 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.851316 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9l7\" (UniqueName: \"kubernetes.io/projected/1a786707-9324-49d4-89dc-decd70a96ba5-kube-api-access-sc9l7\") pod \"payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq\" (UID: \"1a786707-9324-49d4-89dc-decd70a96ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 00:47:21.854796 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.851492 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a786707-9324-49d4-89dc-decd70a96ba5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq\" (UID: \"1a786707-9324-49d4-89dc-decd70a96ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 00:47:21.863481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.863401 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs"] Nov 03 00:47:21.868423 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.868389 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 00:47:21.914835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.914806 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs"] Nov 03 00:47:21.952518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.952440 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9l7\" (UniqueName: \"kubernetes.io/projected/1a786707-9324-49d4-89dc-decd70a96ba5-kube-api-access-sc9l7\") pod \"payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq\" (UID: \"1a786707-9324-49d4-89dc-decd70a96ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 00:47:21.952693 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.952553 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a786707-9324-49d4-89dc-decd70a96ba5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq\" (UID: \"1a786707-9324-49d4-89dc-decd70a96ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 00:47:21.953538 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:21.953514 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a786707-9324-49d4-89dc-decd70a96ba5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq\" (UID: \"1a786707-9324-49d4-89dc-decd70a96ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 00:47:22.009707 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:22.009669 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9l7\" (UniqueName: \"kubernetes.io/projected/1a786707-9324-49d4-89dc-decd70a96ba5-kube-api-access-sc9l7\") pod \"payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq\" (UID: \"1a786707-9324-49d4-89dc-decd70a96ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 00:47:22.053737 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:22.053697 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29794f37-0e98-48b3-aa42-dd12a31f9280-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs\" (UID: \"29794f37-0e98-48b3-aa42-dd12a31f9280\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 00:47:22.053907 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:22.053829 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9dd\" (UniqueName: \"kubernetes.io/projected/29794f37-0e98-48b3-aa42-dd12a31f9280-kube-api-access-vf9dd\") pod \"payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs\" (UID: \"29794f37-0e98-48b3-aa42-dd12a31f9280\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 00:47:22.154846 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:22.154814 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9dd\" (UniqueName: \"kubernetes.io/projected/29794f37-0e98-48b3-aa42-dd12a31f9280-kube-api-access-vf9dd\") pod \"payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs\" (UID: \"29794f37-0e98-48b3-aa42-dd12a31f9280\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 00:47:22.155049 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:22.154980 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29794f37-0e98-48b3-aa42-dd12a31f9280-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs\" (UID: \"29794f37-0e98-48b3-aa42-dd12a31f9280\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 00:47:22.155597 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:22.155579 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29794f37-0e98-48b3-aa42-dd12a31f9280-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs\" (UID: \"29794f37-0e98-48b3-aa42-dd12a31f9280\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 00:47:22.211395 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:22.211325 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9dd\" (UniqueName: \"kubernetes.io/projected/29794f37-0e98-48b3-aa42-dd12a31f9280-kube-api-access-vf9dd\") pod \"payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs\" (UID: \"29794f37-0e98-48b3-aa42-dd12a31f9280\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 00:47:22.309562 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:22.309529 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 00:47:22.484347 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:22.484265 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 00:47:22.684184 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:22.684157 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq"] Nov 03 00:47:22.685111 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:47:22.685082 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a786707_9324_49d4_89dc_decd70a96ba5.slice/crio-307fe70d7a9874adb432950a0b050a8a12ea7df038dac84fc2489d7a6adfba46 WatchSource:0}: Error finding container 307fe70d7a9874adb432950a0b050a8a12ea7df038dac84fc2489d7a6adfba46: Status 404 returned error can't find the container with id 307fe70d7a9874adb432950a0b050a8a12ea7df038dac84fc2489d7a6adfba46 Nov 03 00:47:22.859744 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:22.859720 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs"] Nov 03 00:47:22.861307 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:47:22.861281 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29794f37_0e98_48b3_aa42_dd12a31f9280.slice/crio-6011f7a87ec7aaa270f760b184610c87db9dae4d1b6b34365816a798dc887857 WatchSource:0}: Error finding container 6011f7a87ec7aaa270f760b184610c87db9dae4d1b6b34365816a798dc887857: Status 404 returned error can't find the container with id 6011f7a87ec7aaa270f760b184610c87db9dae4d1b6b34365816a798dc887857 Nov 03 00:47:23.271703 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.271663 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" event={"ID":"1a786707-9324-49d4-89dc-decd70a96ba5","Type":"ContainerStarted","Data":"d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0"} Nov 03 00:47:23.271982 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.271729 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" event={"ID":"1a786707-9324-49d4-89dc-decd70a96ba5","Type":"ContainerStarted","Data":"307fe70d7a9874adb432950a0b050a8a12ea7df038dac84fc2489d7a6adfba46"} Nov 03 00:47:23.271982 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.271764 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 00:47:23.273481 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.273447 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" event={"ID":"29794f37-0e98-48b3-aa42-dd12a31f9280","Type":"ContainerStarted","Data":"21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89"} Nov 03 00:47:23.273591 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.273505 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" event={"ID":"29794f37-0e98-48b3-aa42-dd12a31f9280","Type":"ContainerStarted","Data":"6011f7a87ec7aaa270f760b184610c87db9dae4d1b6b34365816a798dc887857"} Nov 03 00:47:23.273591 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.273532 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 00:47:23.386201 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.386147 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" podStartSLOduration=2.386133507 podStartE2EDuration="2.386133507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:23.382918616 +0000 UTC m=+2526.666476288" watchObservedRunningTime="2026-11-03 00:47:23.386133507 +0000 UTC m=+2526.669691179" Nov 03 00:47:23.499962 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.499876 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" podStartSLOduration=2.49985707 podStartE2EDuration="2.49985707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:23.494995514 +0000 UTC m=+2526.778553191" watchObservedRunningTime="2026-11-03 00:47:23.49985707 +0000 UTC m=+2526.783414745" Nov 03 00:47:23.935264 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.935224 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9"] Nov 03 00:47:23.939474 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.939453 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:23.980118 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.980094 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad821b12-e889-4259-894c-a7f4e294b33e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:23.980258 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.980237 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:23.980314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.980302 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:23.980407 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.980394 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kst2b\" (UniqueName: \"kubernetes.io/projected/ad821b12-e889-4259-894c-a7f4e294b33e-kube-api-access-kst2b\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:23.999787 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:23.999748 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9"] Nov 03 00:47:24.044787 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:24.044751 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-904a34-serving-cert\"" Nov 03 00:47:24.081713 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:24.081682 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:24.081854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:24.081787 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kst2b\" (UniqueName: \"kubernetes.io/projected/ad821b12-e889-4259-894c-a7f4e294b33e-kube-api-access-kst2b\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:24.081854 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:24.081845 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad821b12-e889-4259-894c-a7f4e294b33e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:24.081941 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:47:24.081859 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-serving-cert: secret "maas-api-e2e-shared-b-w1-904a34-serving-cert" not found Nov 03 00:47:24.081941 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:47:24.081924 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-maas-api-tls podName:ad821b12-e889-4259-894c-a7f4e294b33e nodeName:}" failed. No retries permitted until 2026-11-03 00:47:24.581906582 +0000 UTC m=+2527.865464251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-maas-api-tls") pod "maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" (UID: "ad821b12-e889-4259-894c-a7f4e294b33e") : secret "maas-api-e2e-shared-b-w1-904a34-serving-cert" not found Nov 03 00:47:24.082035 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:24.081988 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:24.082397 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:24.082369 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad821b12-e889-4259-894c-a7f4e294b33e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:24.084376 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:24.084353 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:24.140094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:24.140063 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kst2b\" (UniqueName: \"kubernetes.io/projected/ad821b12-e889-4259-894c-a7f4e294b33e-kube-api-access-kst2b\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:24.601743 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:24.601713 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:24.604234 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:24.604208 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:24.853382 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:24.853310 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:25.229851 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:47:25.229823 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad821b12_e889_4259_894c_a7f4e294b33e.slice/crio-1b86459ad32a7abad39fcbad432286787b9d9f8ff3cf0d44c3bb990dbec656d7 WatchSource:0}: Error finding container 1b86459ad32a7abad39fcbad432286787b9d9f8ff3cf0d44c3bb990dbec656d7: Status 404 returned error can't find the container with id 1b86459ad32a7abad39fcbad432286787b9d9f8ff3cf0d44c3bb990dbec656d7 Nov 03 00:47:25.263716 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:25.263686 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9"] Nov 03 00:47:25.287220 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:25.287190 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" event={"ID":"ad821b12-e889-4259-894c-a7f4e294b33e","Type":"ContainerStarted","Data":"1b86459ad32a7abad39fcbad432286787b9d9f8ff3cf0d44c3bb990dbec656d7"} Nov 03 00:47:26.294620 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:26.294588 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" event={"ID":"ad821b12-e889-4259-894c-a7f4e294b33e","Type":"ContainerStarted","Data":"39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750"} Nov 03 00:47:26.295017 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:26.294654 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:26.409482 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:26.409425 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" podStartSLOduration=2.528249707 podStartE2EDuration="3.409410636s" totalImagesPullingTime="881.160929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:26.408845769 +0000 UTC m=+2529.692403439" watchObservedRunningTime="2026-11-03 00:47:26.409410636 +0000 UTC m=+2529.692968309" Nov 03 00:47:28.665125 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:47:28.665089 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:47:32.307017 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:32.306989 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 00:47:34.282863 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:34.282831 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 00:47:34.283291 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:47:34.282893 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 00:47:58.654168 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:47:58.654122 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:48:21.803637 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:21.803595 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc"] Nov 03 00:48:21.807628 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:21.807605 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:21.861163 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:21.861134 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-20f0a3-serving-cert\"" Nov 03 00:48:21.865669 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:21.865646 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc"] Nov 03 00:48:21.902653 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:21.902615 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:21.902842 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:21.902822 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjkl\" (UniqueName: \"kubernetes.io/projected/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-kube-api-access-drjkl\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:21.902969 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:21.902950 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:21.903036 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:21.903006 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:22.003820 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:22.003764 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:22.003987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:22.003892 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjkl\" (UniqueName: \"kubernetes.io/projected/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-kube-api-access-drjkl\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:22.003987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:22.003940 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:22.003987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:22.003971 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:22.004173 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:48:22.004051 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-serving-cert: secret "maas-api-e2e-shared-b-w4-20f0a3-serving-cert" not found Nov 03 00:48:22.004173 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:48:22.004128 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-maas-api-tls podName:bff46dd8-5eb0-4114-8bdc-13964d0c0b23 nodeName:}" failed. No retries permitted until 2026-11-03 00:48:22.504107642 +0000 UTC m=+2585.787665296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-maas-api-tls") pod "maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" (UID: "bff46dd8-5eb0-4114-8bdc-13964d0c0b23") : secret "maas-api-e2e-shared-b-w4-20f0a3-serving-cert" not found Nov 03 00:48:22.004449 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:22.004428 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:22.006497 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:22.006473 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:22.070910 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:22.070841 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjkl\" (UniqueName: \"kubernetes.io/projected/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-kube-api-access-drjkl\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:22.509308 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:22.509274 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:22.511731 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:22.511712 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:22.720622 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:22.720591 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:23.101906 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:23.101875 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc"] Nov 03 00:48:23.103206 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:48:23.103179 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff46dd8_5eb0_4114_8bdc_13964d0c0b23.slice/crio-4810dcd972d883edde543a1f6f331c8544e09a52e7c98c6bb96bed34208cafc2 WatchSource:0}: Error finding container 4810dcd972d883edde543a1f6f331c8544e09a52e7c98c6bb96bed34208cafc2: Status 404 returned error can't find the container with id 4810dcd972d883edde543a1f6f331c8544e09a52e7c98c6bb96bed34208cafc2 Nov 03 00:48:23.654415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:23.654380 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" event={"ID":"bff46dd8-5eb0-4114-8bdc-13964d0c0b23","Type":"ContainerStarted","Data":"4810dcd972d883edde543a1f6f331c8544e09a52e7c98c6bb96bed34208cafc2"} Nov 03 00:48:24.662441 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:24.662405 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" event={"ID":"bff46dd8-5eb0-4114-8bdc-13964d0c0b23","Type":"ContainerStarted","Data":"4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc"} Nov 03 00:48:24.662898 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:24.662539 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:24.779704 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:24.779651 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" podStartSLOduration=2.982455266 podStartE2EDuration="3.77963613s" totalImagesPullingTime="797.180864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:24.777206134 +0000 UTC m=+2588.060763812" watchObservedRunningTime="2026-11-03 00:48:24.77963613 +0000 UTC m=+2588.063193834" Nov 03 00:48:28.654963 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:48:28.654916 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:48:30.674169 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:48:30.674143 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 00:48:58.654864 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:48:58.654831 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:49:28.654825 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:49:28.654762 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:49:43.679263 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.679185 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q"] Nov 03 00:49:43.683228 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.683212 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.764220 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.764187 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-model-cache\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.764399 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.764225 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsd75\" (UniqueName: \"kubernetes.io/projected/06812e91-a171-4102-a6fa-bacb643db495-kube-api-access-fsd75\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.764399 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.764313 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-dshm\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.764399 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.764376 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-tmp-dir\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.764574 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.764549 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-home\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.764663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.764642 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06812e91-a171-4102-a6fa-bacb643db495-tls-certs\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.785866 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.785833 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c8ce60\"/\"openshift-service-ca.crt\"" Nov 03 00:49:43.785866 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.785854 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c8ce60\"/\"kube-root-ca.crt\"" Nov 03 00:49:43.786046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.785943 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c8ce60\"/\"default-dockercfg-wbmp7\"" Nov 03 00:49:43.786046 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.785979 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c8ce60\"/\"auth-test-model-c8ce60-kserve-self-signed-certs\"" Nov 03 00:49:43.789286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.789269 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q"] Nov 03 00:49:43.866222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.866192 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-model-cache\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.866222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.866223 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsd75\" (UniqueName: \"kubernetes.io/projected/06812e91-a171-4102-a6fa-bacb643db495-kube-api-access-fsd75\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.866484 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.866252 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-dshm\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.866484 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.866306 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-tmp-dir\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.866484 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.866402 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-home\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.866484 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.866461 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06812e91-a171-4102-a6fa-bacb643db495-tls-certs\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.866695 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.866624 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-model-cache\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.866755 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.866693 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-tmp-dir\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.866858 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.866804 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-home\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.868620 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.868601 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-dshm\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.868994 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.868976 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06812e91-a171-4102-a6fa-bacb643db495-tls-certs\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.928787 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.928749 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsd75\" (UniqueName: \"kubernetes.io/projected/06812e91-a171-4102-a6fa-bacb643db495-kube-api-access-fsd75\") pod \"auth-test-model-c8ce60-kserve-75b7948bf-h7c6q\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:43.994716 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:43.994641 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:44.370130 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:44.370104 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q"] Nov 03 00:49:44.371683 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:49:44.371647 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06812e91_a171_4102_a6fa_bacb643db495.slice/crio-3bfb91d38eb05d3410a0f4902cf149575aa5fc8a882887b5085f5fcc4b0a539c WatchSource:0}: Error finding container 3bfb91d38eb05d3410a0f4902cf149575aa5fc8a882887b5085f5fcc4b0a539c: Status 404 returned error can't find the container with id 3bfb91d38eb05d3410a0f4902cf149575aa5fc8a882887b5085f5fcc4b0a539c Nov 03 00:49:45.183472 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:45.183428 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" event={"ID":"06812e91-a171-4102-a6fa-bacb643db495","Type":"ContainerStarted","Data":"42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a"} Nov 03 00:49:45.183472 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:45.183474 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" event={"ID":"06812e91-a171-4102-a6fa-bacb643db495","Type":"ContainerStarted","Data":"3bfb91d38eb05d3410a0f4902cf149575aa5fc8a882887b5085f5fcc4b0a539c"} Nov 03 00:49:45.297349 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:45.297287 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" podStartSLOduration=2.29726373 podStartE2EDuration="2.29726373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:45.295236718 +0000 UTC m=+2668.578794393" watchObservedRunningTime="2026-11-03 00:49:45.29726373 +0000 UTC m=+2668.580821399" Nov 03 00:49:45.995688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:45.995657 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:50.440216 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:50.440180 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l"] Nov 03 00:49:50.440660 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:50.440407 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" containerName="istio-proxy" containerID="cri-o://99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a" gracePeriod=30 Nov 03 00:49:53.995672 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:53.995633 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:54.008399 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:54.008361 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:54.251499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:54.251414 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:49:55.711681 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.711621 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:49:55.810246 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810218 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-data\") pod \"202db0b8-6ff6-4493-a92f-66d68745617d\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " Nov 03 00:49:55.810414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810286 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-kube-api-access-k5xs8\" (UniqueName: \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-kube-api-access-k5xs8\") pod \"202db0b8-6ff6-4493-a92f-66d68745617d\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " Nov 03 00:49:55.810414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810308 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202db0b8-6ff6-4493-a92f-66d68745617d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202db0b8-6ff6-4493-a92f-66d68745617d-istiod-ca-cert\") pod \"202db0b8-6ff6-4493-a92f-66d68745617d\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " Nov 03 00:49:55.810414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810361 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-envoy\") pod \"202db0b8-6ff6-4493-a92f-66d68745617d\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " Nov 03 00:49:55.810414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810388 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-certs\") pod \"202db0b8-6ff6-4493-a92f-66d68745617d\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " Nov 03 00:49:55.810608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810427 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-istio-token\" (UniqueName: \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-istio-token\") pod \"202db0b8-6ff6-4493-a92f-66d68745617d\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " Nov 03 00:49:55.810608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810453 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-socket\") pod \"202db0b8-6ff6-4493-a92f-66d68745617d\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " Nov 03 00:49:55.810717 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810617 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/202db0b8-6ff6-4493-a92f-66d68745617d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202db0b8-6ff6-4493-a92f-66d68745617d-istio-podinfo\") pod \"202db0b8-6ff6-4493-a92f-66d68745617d\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " Nov 03 00:49:55.810717 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810657 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-credential-socket\") pod \"202db0b8-6ff6-4493-a92f-66d68745617d\" (UID: \"202db0b8-6ff6-4493-a92f-66d68745617d\") " Nov 03 00:49:55.810864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810710 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-certs" pod "202db0b8-6ff6-4493-a92f-66d68745617d" (UID: "202db0b8-6ff6-4493-a92f-66d68745617d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:49:55.810864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810716 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202db0b8-6ff6-4493-a92f-66d68745617d-istiod-ca-cert" pod "202db0b8-6ff6-4493-a92f-66d68745617d" (UID: "202db0b8-6ff6-4493-a92f-66d68745617d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:49:55.810864 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810829 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-socket" pod "202db0b8-6ff6-4493-a92f-66d68745617d" (UID: "202db0b8-6ff6-4493-a92f-66d68745617d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:49:55.811043 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.810561 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-data" pod "202db0b8-6ff6-4493-a92f-66d68745617d" (UID: "202db0b8-6ff6-4493-a92f-66d68745617d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:49:55.811043 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.811008 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-credential-socket" pod "202db0b8-6ff6-4493-a92f-66d68745617d" (UID: "202db0b8-6ff6-4493-a92f-66d68745617d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:49:55.811232 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.811211 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.811314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.811238 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.811314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.811253 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202db0b8-6ff6-4493-a92f-66d68745617d-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.811314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.811269 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.811314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.811283 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.812884 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.812841 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-kube-api-access-k5xs8" pod "202db0b8-6ff6-4493-a92f-66d68745617d" (UID: "202db0b8-6ff6-4493-a92f-66d68745617d"). InnerVolumeSpecName "kube-api-access-k5xs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:49:55.813037 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.813016 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/202db0b8-6ff6-4493-a92f-66d68745617d-istio-podinfo" pod "202db0b8-6ff6-4493-a92f-66d68745617d" (UID: "202db0b8-6ff6-4493-a92f-66d68745617d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:49:55.813227 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.813198 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-envoy" pod "202db0b8-6ff6-4493-a92f-66d68745617d" (UID: "202db0b8-6ff6-4493-a92f-66d68745617d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:49:55.813227 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.813220 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-istio-token" pod "202db0b8-6ff6-4493-a92f-66d68745617d" (UID: "202db0b8-6ff6-4493-a92f-66d68745617d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:49:55.912593 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.912518 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202db0b8-6ff6-4493-a92f-66d68745617d-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.912816 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.912799 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5xs8\" (UniqueName: \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-kube-api-access-k5xs8\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.912964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.912942 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202db0b8-6ff6-4493-a92f-66d68745617d-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.912964 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:55.912964 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202db0b8-6ff6-4493-a92f-66d68745617d-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:56.257100 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:56.257063 4304 generic.go:356] "Generic (PLEG): container finished" podID="202db0b8-6ff6-4493-a92f-66d68745617d" containerID="99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a" exitCode=0 Nov 03 00:49:56.257309 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:56.257122 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" Nov 03 00:49:56.257309 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:56.257148 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" event={"ID":"202db0b8-6ff6-4493-a92f-66d68745617d","Type":"ContainerDied","Data":"99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a"} Nov 03 00:49:56.257309 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:56.257201 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l" event={"ID":"202db0b8-6ff6-4493-a92f-66d68745617d","Type":"ContainerDied","Data":"421e877bb6c8cd76083f8312b96bf978365a0f4057d71b5de62caea384cc3858"} Nov 03 00:49:56.257309 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:56.257214 4304 scope.go:122] "RemoveContainer" containerID="99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a" Nov 03 00:49:56.267745 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:56.267726 4304 scope.go:122] "RemoveContainer" containerID="99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a" Nov 03 00:49:56.268045 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:49:56.268027 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a\": container with ID starting with 99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a not found: ID does not exist" containerID="99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a" Nov 03 00:49:56.268113 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:56.268058 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a"} err="failed to get container status \"99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a\": rpc error: code = NotFound desc = could not find container \"99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a\": container with ID starting with 99345f1769fbb63eb85cdeec67384c1966f3aedbf1d16b47d34f478378d65e5a not found: ID does not exist" Nov 03 00:49:56.424550 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:56.424517 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l"] Nov 03 00:49:56.474031 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:56.473985 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-qgt9l"] Nov 03 00:49:56.829734 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:49:56.829697 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" path="/var/lib/kubelet/pods/202db0b8-6ff6-4493-a92f-66d68745617d/volumes" Nov 03 00:49:58.654858 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:49:58.654822 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:50:13.680358 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:13.680320 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw"] Nov 03 00:50:13.680893 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:13.680519 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" podUID="09aa3322-317a-4d12-9d97-76d132c60819" containerName="istio-proxy" containerID="cri-o://f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19" gracePeriod=30 Nov 03 00:50:18.942363 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:18.942333 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:50:19.080531 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.080437 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09aa3322-317a-4d12-9d97-76d132c60819-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09aa3322-317a-4d12-9d97-76d132c60819-istio-podinfo\") pod \"09aa3322-317a-4d12-9d97-76d132c60819\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " Nov 03 00:50:19.080531 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.080522 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-istio-token\" (UniqueName: \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-istio-token\") pod \"09aa3322-317a-4d12-9d97-76d132c60819\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " Nov 03 00:50:19.080788 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.080591 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-certs\") pod \"09aa3322-317a-4d12-9d97-76d132c60819\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " Nov 03 00:50:19.080788 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.080621 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-credential-socket\") pod \"09aa3322-317a-4d12-9d97-76d132c60819\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " Nov 03 00:50:19.080788 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.080657 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09aa3322-317a-4d12-9d97-76d132c60819-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09aa3322-317a-4d12-9d97-76d132c60819-istiod-ca-cert\") pod \"09aa3322-317a-4d12-9d97-76d132c60819\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " Nov 03 00:50:19.080788 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.080719 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-socket\") pod \"09aa3322-317a-4d12-9d97-76d132c60819\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " Nov 03 00:50:19.080788 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.080760 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-kube-api-access-5z849\" (UniqueName: \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-kube-api-access-5z849\") pod \"09aa3322-317a-4d12-9d97-76d132c60819\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " Nov 03 00:50:19.081094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.080805 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-envoy\") pod \"09aa3322-317a-4d12-9d97-76d132c60819\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " Nov 03 00:50:19.081094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.080865 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-data\") pod \"09aa3322-317a-4d12-9d97-76d132c60819\" (UID: \"09aa3322-317a-4d12-9d97-76d132c60819\") " Nov 03 00:50:19.081094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.080910 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-certs" pod "09aa3322-317a-4d12-9d97-76d132c60819" (UID: "09aa3322-317a-4d12-9d97-76d132c60819"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:19.081094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.080920 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-credential-socket" pod "09aa3322-317a-4d12-9d97-76d132c60819" (UID: "09aa3322-317a-4d12-9d97-76d132c60819"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:19.081094 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.081066 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-socket" pod "09aa3322-317a-4d12-9d97-76d132c60819" (UID: "09aa3322-317a-4d12-9d97-76d132c60819"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:19.081515 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.081149 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09aa3322-317a-4d12-9d97-76d132c60819-istiod-ca-cert" pod "09aa3322-317a-4d12-9d97-76d132c60819" (UID: "09aa3322-317a-4d12-9d97-76d132c60819"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:19.081515 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.081276 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-data" pod "09aa3322-317a-4d12-9d97-76d132c60819" (UID: "09aa3322-317a-4d12-9d97-76d132c60819"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:19.081634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.081524 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.081634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.081541 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.081634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.081553 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.081634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.081566 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09aa3322-317a-4d12-9d97-76d132c60819-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.081634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.081577 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.083013 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.082992 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09aa3322-317a-4d12-9d97-76d132c60819-istio-podinfo" pod "09aa3322-317a-4d12-9d97-76d132c60819" (UID: "09aa3322-317a-4d12-9d97-76d132c60819"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:50:19.083110 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.083018 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-istio-token" pod "09aa3322-317a-4d12-9d97-76d132c60819" (UID: "09aa3322-317a-4d12-9d97-76d132c60819"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:19.083488 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.083468 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-envoy" pod "09aa3322-317a-4d12-9d97-76d132c60819" (UID: "09aa3322-317a-4d12-9d97-76d132c60819"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:19.083488 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.083477 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-kube-api-access-5z849" pod "09aa3322-317a-4d12-9d97-76d132c60819" (UID: "09aa3322-317a-4d12-9d97-76d132c60819"). InnerVolumeSpecName "kube-api-access-5z849". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:19.182688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.182654 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z849\" (UniqueName: \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-kube-api-access-5z849\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.182688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.182684 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09aa3322-317a-4d12-9d97-76d132c60819-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.182688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.182693 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09aa3322-317a-4d12-9d97-76d132c60819-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.182959 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.182702 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09aa3322-317a-4d12-9d97-76d132c60819-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:19.398596 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.398562 4304 generic.go:356] "Generic (PLEG): container finished" podID="09aa3322-317a-4d12-9d97-76d132c60819" containerID="f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19" exitCode=0 Nov 03 00:50:19.398766 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.398615 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" Nov 03 00:50:19.398766 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.398652 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" event={"ID":"09aa3322-317a-4d12-9d97-76d132c60819","Type":"ContainerDied","Data":"f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19"} Nov 03 00:50:19.398766 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.398713 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw" event={"ID":"09aa3322-317a-4d12-9d97-76d132c60819","Type":"ContainerDied","Data":"4c1eda88ee9a44781b26bef0c2303ac91c386284fce19e02543b17959e0b8081"} Nov 03 00:50:19.398766 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.398740 4304 scope.go:122] "RemoveContainer" containerID="f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19" Nov 03 00:50:19.412038 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.411935 4304 scope.go:122] "RemoveContainer" containerID="f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19" Nov 03 00:50:19.412400 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:50:19.412380 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19\": container with ID starting with f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19 not found: ID does not exist" containerID="f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19" Nov 03 00:50:19.412483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.412410 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19"} err="failed to get container status \"f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19\": rpc error: code = NotFound desc = could not find container \"f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19\": container with ID starting with f2c8170a3d6de403f4c5aa00dd893cb2a929e00903b46b6f9eaeaf966adc6c19 not found: ID does not exist" Nov 03 00:50:19.562840 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.562809 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw"] Nov 03 00:50:19.615571 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:19.615538 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-p4kcw"] Nov 03 00:50:20.830041 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:20.830009 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09aa3322-317a-4d12-9d97-76d132c60819" path="/var/lib/kubelet/pods/09aa3322-317a-4d12-9d97-76d132c60819/volumes" Nov 03 00:50:28.654996 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:50:28.654956 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:50:35.790679 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.790641 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn"] Nov 03 00:50:35.791263 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.791245 4304 cpu_manager.go:414] "containerMap: removing container" podUID="09aa3322-317a-4d12-9d97-76d132c60819" containerName="istio-proxy" Nov 03 00:50:35.791305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.791264 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09aa3322-317a-4d12-9d97-76d132c60819" containerName="istio-proxy" Nov 03 00:50:35.791305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.791285 4304 cpu_manager.go:414] "containerMap: removing container" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" containerName="istio-proxy" Nov 03 00:50:35.791305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.791290 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" containerName="istio-proxy" Nov 03 00:50:35.791411 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.791400 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="09aa3322-317a-4d12-9d97-76d132c60819" containerName="istio-proxy" Nov 03 00:50:35.791445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.791415 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="202db0b8-6ff6-4493-a92f-66d68745617d" containerName="istio-proxy" Nov 03 00:50:35.796111 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.796094 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.874756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.874721 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-dshm\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.874958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.874761 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gv8z\" (UniqueName: \"kubernetes.io/projected/72d98d2f-4f35-49f9-9f4a-34986bb3799f-kube-api-access-4gv8z\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.874958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.874853 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-home\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.875092 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.875009 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-model-cache\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.875150 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.875131 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tls-certs\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.875233 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.875218 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tmp-dir\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.897422 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.897390 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-904a34\"/\"kube-root-ca.crt\"" Nov 03 00:50:35.897422 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.897408 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-904a34\"/\"default-dockercfg-txv2k\"" Nov 03 00:50:35.897585 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.897424 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-904a34\"/\"auth-test-model-904a34-kserve-self-signed-certs\"" Nov 03 00:50:35.897585 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.897433 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-904a34\"/\"openshift-service-ca.crt\"" Nov 03 00:50:35.900719 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.900703 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn"] Nov 03 00:50:35.976655 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.976623 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-dshm\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.976896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.976661 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gv8z\" (UniqueName: \"kubernetes.io/projected/72d98d2f-4f35-49f9-9f4a-34986bb3799f-kube-api-access-4gv8z\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.976896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.976688 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-home\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.976896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.976742 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-model-cache\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.976896 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.976839 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tls-certs\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.977132 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.976929 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tmp-dir\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.977286 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.977262 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-home\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.977358 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.977286 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tmp-dir\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.977539 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.977518 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-model-cache\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.979206 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.979184 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-dshm\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:35.979574 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:35.979555 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tls-certs\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:36.033827 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:36.033794 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gv8z\" (UniqueName: \"kubernetes.io/projected/72d98d2f-4f35-49f9-9f4a-34986bb3799f-kube-api-access-4gv8z\") pod \"auth-test-model-904a34-kserve-65f57b6cfc-8gqvn\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:36.108568 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:36.108488 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:36.483888 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:36.483860 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn"] Nov 03 00:50:36.484682 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:50:36.484653 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72d98d2f_4f35_49f9_9f4a_34986bb3799f.slice/crio-8f6d57c9ac3e12eeb98f321b9dcc684e535d461f2382c7e286bffc9dbaa9e2c3 WatchSource:0}: Error finding container 8f6d57c9ac3e12eeb98f321b9dcc684e535d461f2382c7e286bffc9dbaa9e2c3: Status 404 returned error can't find the container with id 8f6d57c9ac3e12eeb98f321b9dcc684e535d461f2382c7e286bffc9dbaa9e2c3 Nov 03 00:50:36.507367 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:36.507341 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" event={"ID":"72d98d2f-4f35-49f9-9f4a-34986bb3799f","Type":"ContainerStarted","Data":"8f6d57c9ac3e12eeb98f321b9dcc684e535d461f2382c7e286bffc9dbaa9e2c3"} Nov 03 00:50:37.514307 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:37.514270 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" event={"ID":"72d98d2f-4f35-49f9-9f4a-34986bb3799f","Type":"ContainerStarted","Data":"9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad"} Nov 03 00:50:37.629669 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:37.629615 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" podStartSLOduration=2.629602297 podStartE2EDuration="2.629602297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:37.627100948 +0000 UTC m=+2720.910658622" watchObservedRunningTime="2026-11-03 00:50:37.629602297 +0000 UTC m=+2720.913159977" Nov 03 00:50:38.109488 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:38.109452 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:46.109231 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:46.109194 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:46.121370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:46.121346 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:46.584068 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:46.584034 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:50:58.651904 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:58.651870 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz"] Nov 03 00:50:58.652749 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:50:58.652718 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" podUID="076018d2-c750-4a75-ad71-acf5d701136a" containerName="istio-proxy" containerID="cri-o://f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce" gracePeriod=30 Nov 03 00:50:58.657386 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:50:58.657351 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:51:03.928491 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:03.928430 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:51:04.007608 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.007579 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-certs\") pod \"076018d2-c750-4a75-ad71-acf5d701136a\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " Nov 03 00:51:04.007769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.007615 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/076018d2-c750-4a75-ad71-acf5d701136a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076018d2-c750-4a75-ad71-acf5d701136a-istio-podinfo\") pod \"076018d2-c750-4a75-ad71-acf5d701136a\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " Nov 03 00:51:04.007769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.007698 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-kube-api-access-tlw28\" (UniqueName: \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-kube-api-access-tlw28\") pod \"076018d2-c750-4a75-ad71-acf5d701136a\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " Nov 03 00:51:04.007769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.007725 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-credential-socket\") pod \"076018d2-c750-4a75-ad71-acf5d701136a\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " Nov 03 00:51:04.007917 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.007799 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-data\") pod \"076018d2-c750-4a75-ad71-acf5d701136a\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " Nov 03 00:51:04.007917 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.007866 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076018d2-c750-4a75-ad71-acf5d701136a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076018d2-c750-4a75-ad71-acf5d701136a-istiod-ca-cert\") pod \"076018d2-c750-4a75-ad71-acf5d701136a\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " Nov 03 00:51:04.007917 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.007868 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-certs" pod "076018d2-c750-4a75-ad71-acf5d701136a" (UID: "076018d2-c750-4a75-ad71-acf5d701136a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:51:04.008045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.007920 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-socket\") pod \"076018d2-c750-4a75-ad71-acf5d701136a\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " Nov 03 00:51:04.008045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.008003 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-envoy\") pod \"076018d2-c750-4a75-ad71-acf5d701136a\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " Nov 03 00:51:04.008045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.008025 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-istio-token\" (UniqueName: \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-istio-token\") pod \"076018d2-c750-4a75-ad71-acf5d701136a\" (UID: \"076018d2-c750-4a75-ad71-acf5d701136a\") " Nov 03 00:51:04.008045 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.008032 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-data" pod "076018d2-c750-4a75-ad71-acf5d701136a" (UID: "076018d2-c750-4a75-ad71-acf5d701136a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:51:04.008256 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.008054 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-credential-socket" pod "076018d2-c750-4a75-ad71-acf5d701136a" (UID: "076018d2-c750-4a75-ad71-acf5d701136a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:51:04.008317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.008281 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-socket" pod "076018d2-c750-4a75-ad71-acf5d701136a" (UID: "076018d2-c750-4a75-ad71-acf5d701136a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:51:04.008317 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.008299 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076018d2-c750-4a75-ad71-acf5d701136a-istiod-ca-cert" pod "076018d2-c750-4a75-ad71-acf5d701136a" (UID: "076018d2-c750-4a75-ad71-acf5d701136a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:51:04.008490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.008469 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:04.008558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.008496 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:04.008558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.008510 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:04.008558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.008518 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076018d2-c750-4a75-ad71-acf5d701136a-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:04.008558 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.008526 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:04.010151 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.010126 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-kube-api-access-tlw28" pod "076018d2-c750-4a75-ad71-acf5d701136a" (UID: "076018d2-c750-4a75-ad71-acf5d701136a"). InnerVolumeSpecName "kube-api-access-tlw28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:51:04.010290 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.010267 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-envoy" pod "076018d2-c750-4a75-ad71-acf5d701136a" (UID: "076018d2-c750-4a75-ad71-acf5d701136a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:51:04.010402 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.010386 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-istio-token" pod "076018d2-c750-4a75-ad71-acf5d701136a" (UID: "076018d2-c750-4a75-ad71-acf5d701136a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:51:04.010447 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.010377 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/076018d2-c750-4a75-ad71-acf5d701136a-istio-podinfo" pod "076018d2-c750-4a75-ad71-acf5d701136a" (UID: "076018d2-c750-4a75-ad71-acf5d701136a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:51:04.109541 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.109506 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlw28\" (UniqueName: \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-kube-api-access-tlw28\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:04.109830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.109756 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076018d2-c750-4a75-ad71-acf5d701136a-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:04.109830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.109799 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076018d2-c750-4a75-ad71-acf5d701136a-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:04.109830 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.109814 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076018d2-c750-4a75-ad71-acf5d701136a-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:04.688348 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.688314 4304 generic.go:356] "Generic (PLEG): container finished" podID="076018d2-c750-4a75-ad71-acf5d701136a" containerID="f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce" exitCode=0 Nov 03 00:51:04.688523 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.688381 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" Nov 03 00:51:04.688523 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.688392 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" event={"ID":"076018d2-c750-4a75-ad71-acf5d701136a","Type":"ContainerDied","Data":"f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce"} Nov 03 00:51:04.688523 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.688463 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz" event={"ID":"076018d2-c750-4a75-ad71-acf5d701136a","Type":"ContainerDied","Data":"a817cb77831cd86da235648efde0aa44a1f11b3f9c1ac66479794fb5404129c6"} Nov 03 00:51:04.688523 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.688485 4304 scope.go:122] "RemoveContainer" containerID="f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce" Nov 03 00:51:04.699538 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.699521 4304 scope.go:122] "RemoveContainer" containerID="f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce" Nov 03 00:51:04.699881 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:51:04.699839 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce\": container with ID starting with f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce not found: ID does not exist" containerID="f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce" Nov 03 00:51:04.699881 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.699866 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce"} err="failed to get container status \"f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce\": rpc error: code = NotFound desc = could not find container \"f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce\": container with ID starting with f22bdff23ddae30d1b54c300ace7db896025c7f095b3b368adcede1a8efadcce not found: ID does not exist" Nov 03 00:51:04.903407 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.903368 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz"] Nov 03 00:51:04.951518 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:04.951447 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-4qrgz"] Nov 03 00:51:06.831218 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:06.831186 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076018d2-c750-4a75-ad71-acf5d701136a" path="/var/lib/kubelet/pods/076018d2-c750-4a75-ad71-acf5d701136a/volumes" Nov 03 00:51:28.654237 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:51:28.654196 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:51:50.486708 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.486679 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r"] Nov 03 00:51:50.489295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.487272 4304 cpu_manager.go:414] "containerMap: removing container" podUID="076018d2-c750-4a75-ad71-acf5d701136a" containerName="istio-proxy" Nov 03 00:51:50.489295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.487284 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076018d2-c750-4a75-ad71-acf5d701136a" containerName="istio-proxy" Nov 03 00:51:50.489295 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.487373 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="076018d2-c750-4a75-ad71-acf5d701136a" containerName="istio-proxy" Nov 03 00:51:50.490663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.490646 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.593528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.593500 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-328e1b\"/\"kube-root-ca.crt\"" Nov 03 00:51:50.593728 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.593706 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-328e1b\"/\"openshift-service-ca.crt\"" Nov 03 00:51:50.593848 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.593801 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-328e1b\"/\"auth-test-model-328e1b-kserve-self-signed-certs\"" Nov 03 00:51:50.593908 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.593879 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-328e1b\"/\"default-dockercfg-j4b4d\"" Nov 03 00:51:50.596442 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.596423 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r"] Nov 03 00:51:50.648670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.648641 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-dshm\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.648920 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.648684 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-tmp-dir\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.648920 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.648715 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4jn\" (UniqueName: \"kubernetes.io/projected/b5339273-9767-488e-9064-bb7c35d6b4db-kube-api-access-4d4jn\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.649012 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.648941 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-home\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.649061 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.649006 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5339273-9767-488e-9064-bb7c35d6b4db-tls-certs\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.649177 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.649159 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-model-cache\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.750318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.750239 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-dshm\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.750318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.750279 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-tmp-dir\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.750318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.750295 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4jn\" (UniqueName: \"kubernetes.io/projected/b5339273-9767-488e-9064-bb7c35d6b4db-kube-api-access-4d4jn\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.750318 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.750319 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-home\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.750650 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.750356 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5339273-9767-488e-9064-bb7c35d6b4db-tls-certs\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.750650 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.750447 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-model-cache\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.750789 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.750742 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-tmp-dir\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.750845 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.750801 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-home\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.750845 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.750826 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-model-cache\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.752704 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.752685 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-dshm\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.752944 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.752927 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5339273-9767-488e-9064-bb7c35d6b4db-tls-certs\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:50.807305 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:50.807277 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4jn\" (UniqueName: \"kubernetes.io/projected/b5339273-9767-488e-9064-bb7c35d6b4db-kube-api-access-4d4jn\") pod \"auth-test-model-328e1b-kserve-66986df545-r5j9r\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:51.103649 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:51.103565 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:51.477314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:51.477287 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r"] Nov 03 00:51:51.478419 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:51:51.478391 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5339273_9767_488e_9064_bb7c35d6b4db.slice/crio-52a6be42afbef07ee9f8342d8f02aa55a48e483c20a5163b7a9c63d96affeee6 WatchSource:0}: Error finding container 52a6be42afbef07ee9f8342d8f02aa55a48e483c20a5163b7a9c63d96affeee6: Status 404 returned error can't find the container with id 52a6be42afbef07ee9f8342d8f02aa55a48e483c20a5163b7a9c63d96affeee6 Nov 03 00:51:51.978503 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:51.978470 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" event={"ID":"b5339273-9767-488e-9064-bb7c35d6b4db","Type":"ContainerStarted","Data":"4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9"} Nov 03 00:51:51.978503 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:51.978508 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" event={"ID":"b5339273-9767-488e-9064-bb7c35d6b4db","Type":"ContainerStarted","Data":"52a6be42afbef07ee9f8342d8f02aa55a48e483c20a5163b7a9c63d96affeee6"} Nov 03 00:51:52.090978 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:52.090919 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" podStartSLOduration=2.090902246 podStartE2EDuration="2.090902246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:52.090661769 +0000 UTC m=+2795.374219442" watchObservedRunningTime="2026-11-03 00:51:52.090902246 +0000 UTC m=+2795.374459917" Nov 03 00:51:52.104708 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:51:52.104683 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:51:58.654828 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:51:58.654790 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:52:01.104208 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:01.104174 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:52:01.116640 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:01.116619 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:52:02.051571 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:02.051544 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:52:28.655396 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:52:28.655353 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:52:28.657263 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:28.657241 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc"] Nov 03 00:52:28.657478 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:28.657457 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerName="istio-proxy" containerID="cri-o://1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616" gracePeriod=30 Nov 03 00:52:33.912251 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.912226 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:52:33.951987 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.951914 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-podinfo\") pod \"8dd16683-a7b5-4f26-8a6f-2a3160558426\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " Nov 03 00:52:33.952127 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.951986 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-socket\") pod \"8dd16683-a7b5-4f26-8a6f-2a3160558426\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " Nov 03 00:52:33.952127 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952018 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-kube-api-access-w2pf2\" (UniqueName: \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-kube-api-access-w2pf2\") pod \"8dd16683-a7b5-4f26-8a6f-2a3160558426\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " Nov 03 00:52:33.952127 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952064 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-certs\") pod \"8dd16683-a7b5-4f26-8a6f-2a3160558426\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " Nov 03 00:52:33.952127 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952124 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-envoy\") pod \"8dd16683-a7b5-4f26-8a6f-2a3160558426\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " Nov 03 00:52:33.952354 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952197 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-data\") pod \"8dd16683-a7b5-4f26-8a6f-2a3160558426\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " Nov 03 00:52:33.952354 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952229 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd16683-a7b5-4f26-8a6f-2a3160558426-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd16683-a7b5-4f26-8a6f-2a3160558426-istiod-ca-cert\") pod \"8dd16683-a7b5-4f26-8a6f-2a3160558426\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " Nov 03 00:52:33.952354 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952255 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-token\" (UniqueName: \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-token\") pod \"8dd16683-a7b5-4f26-8a6f-2a3160558426\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " Nov 03 00:52:33.952354 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952307 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-credential-socket\") pod \"8dd16683-a7b5-4f26-8a6f-2a3160558426\" (UID: \"8dd16683-a7b5-4f26-8a6f-2a3160558426\") " Nov 03 00:52:33.952354 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952306 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-socket" pod "8dd16683-a7b5-4f26-8a6f-2a3160558426" (UID: "8dd16683-a7b5-4f26-8a6f-2a3160558426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:33.952752 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952651 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-data" pod "8dd16683-a7b5-4f26-8a6f-2a3160558426" (UID: "8dd16683-a7b5-4f26-8a6f-2a3160558426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:33.952928 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952788 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:33.952928 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952806 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:33.952928 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.952910 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-certs" pod "8dd16683-a7b5-4f26-8a6f-2a3160558426" (UID: "8dd16683-a7b5-4f26-8a6f-2a3160558426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:33.953168 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.953142 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd16683-a7b5-4f26-8a6f-2a3160558426-istiod-ca-cert" pod "8dd16683-a7b5-4f26-8a6f-2a3160558426" (UID: "8dd16683-a7b5-4f26-8a6f-2a3160558426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:33.953323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.953209 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-credential-socket" pod "8dd16683-a7b5-4f26-8a6f-2a3160558426" (UID: "8dd16683-a7b5-4f26-8a6f-2a3160558426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:33.954503 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.954470 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-podinfo" pod "8dd16683-a7b5-4f26-8a6f-2a3160558426" (UID: "8dd16683-a7b5-4f26-8a6f-2a3160558426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:52:33.954602 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.954580 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-token" pod "8dd16683-a7b5-4f26-8a6f-2a3160558426" (UID: "8dd16683-a7b5-4f26-8a6f-2a3160558426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:33.954877 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.954852 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-kube-api-access-w2pf2" pod "8dd16683-a7b5-4f26-8a6f-2a3160558426" (UID: "8dd16683-a7b5-4f26-8a6f-2a3160558426"). InnerVolumeSpecName "kube-api-access-w2pf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:33.954986 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:33.954915 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-envoy" pod "8dd16683-a7b5-4f26-8a6f-2a3160558426" (UID: "8dd16683-a7b5-4f26-8a6f-2a3160558426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:34.054152 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.054124 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:34.054152 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.054149 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2pf2\" (UniqueName: \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-kube-api-access-w2pf2\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:34.054323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.054160 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:34.054323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.054168 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:34.054323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.054176 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd16683-a7b5-4f26-8a6f-2a3160558426-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:34.054323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.054184 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd16683-a7b5-4f26-8a6f-2a3160558426-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:34.054323 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.054192 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd16683-a7b5-4f26-8a6f-2a3160558426-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:34.241253 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.241155 4304 generic.go:356] "Generic (PLEG): container finished" podID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerID="1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616" exitCode=0 Nov 03 00:52:34.241253 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.241212 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" event={"ID":"8dd16683-a7b5-4f26-8a6f-2a3160558426","Type":"ContainerDied","Data":"1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616"} Nov 03 00:52:34.241253 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.241221 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" Nov 03 00:52:34.241507 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.241295 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc" event={"ID":"8dd16683-a7b5-4f26-8a6f-2a3160558426","Type":"ContainerDied","Data":"c5712a1541d13033c6a3f3c209dcebe36d8d72d9fecf841939208f49d6076360"} Nov 03 00:52:34.241507 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.241327 4304 scope.go:122] "RemoveContainer" containerID="1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616" Nov 03 00:52:34.253349 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.253285 4304 scope.go:122] "RemoveContainer" containerID="1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616" Nov 03 00:52:34.253639 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:52:34.253608 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616\": container with ID starting with 1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616 not found: ID does not exist" containerID="1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616" Nov 03 00:52:34.253714 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.253646 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616"} err="failed to get container status \"1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616\": rpc error: code = NotFound desc = could not find container \"1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616\": container with ID starting with 1670799c3657bc99c9f5860ef71b615d860ccfe2a31c7cf82d4784bde2b49616 not found: ID does not exist" Nov 03 00:52:34.405551 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.405515 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc"] Nov 03 00:52:34.408408 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.408373 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mgjhc"] Nov 03 00:52:34.830541 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:52:34.830507 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" path="/var/lib/kubelet/pods/8dd16683-a7b5-4f26-8a6f-2a3160558426/volumes" Nov 03 00:52:58.655690 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:52:58.655646 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:53:28.659612 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:53:28.659563 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:53:28.773881 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:28.773816 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc"] Nov 03 00:53:28.774182 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:28.774097 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" podUID="2a413a39-1efa-4c79-8a34-d40431400818" containerName="istio-proxy" containerID="cri-o://ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997" gracePeriod=30 Nov 03 00:53:34.031623 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.031600 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:53:34.129483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.129399 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-data\") pod \"2a413a39-1efa-4c79-8a34-d40431400818\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " Nov 03 00:53:34.129483 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.129465 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-envoy\") pod \"2a413a39-1efa-4c79-8a34-d40431400818\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " Nov 03 00:53:34.129690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.129514 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-kube-api-access-vqp26\" (UniqueName: \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-kube-api-access-vqp26\") pod \"2a413a39-1efa-4c79-8a34-d40431400818\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " Nov 03 00:53:34.129690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.129579 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a413a39-1efa-4c79-8a34-d40431400818-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a413a39-1efa-4c79-8a34-d40431400818-istiod-ca-cert\") pod \"2a413a39-1efa-4c79-8a34-d40431400818\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " Nov 03 00:53:34.129690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.129670 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a413a39-1efa-4c79-8a34-d40431400818-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a413a39-1efa-4c79-8a34-d40431400818-istio-podinfo\") pod \"2a413a39-1efa-4c79-8a34-d40431400818\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " Nov 03 00:53:34.129690 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.129667 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-data" pod "2a413a39-1efa-4c79-8a34-d40431400818" (UID: "2a413a39-1efa-4c79-8a34-d40431400818"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.129951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.129757 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-socket\") pod \"2a413a39-1efa-4c79-8a34-d40431400818\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " Nov 03 00:53:34.129951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.129825 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-credential-socket\") pod \"2a413a39-1efa-4c79-8a34-d40431400818\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " Nov 03 00:53:34.129951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.129858 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-certs\") pod \"2a413a39-1efa-4c79-8a34-d40431400818\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " Nov 03 00:53:34.129951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.129921 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-istio-token\" (UniqueName: \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-istio-token\") pod \"2a413a39-1efa-4c79-8a34-d40431400818\" (UID: \"2a413a39-1efa-4c79-8a34-d40431400818\") " Nov 03 00:53:34.130176 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.130113 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-socket" pod "2a413a39-1efa-4c79-8a34-d40431400818" (UID: "2a413a39-1efa-4c79-8a34-d40431400818"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.130176 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.130049 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a413a39-1efa-4c79-8a34-d40431400818-istiod-ca-cert" pod "2a413a39-1efa-4c79-8a34-d40431400818" (UID: "2a413a39-1efa-4c79-8a34-d40431400818"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:34.130573 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.130443 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a413a39-1efa-4c79-8a34-d40431400818-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.130573 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.130502 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.130573 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.130519 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.130573 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.130537 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-certs" pod "2a413a39-1efa-4c79-8a34-d40431400818" (UID: "2a413a39-1efa-4c79-8a34-d40431400818"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.131267 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.131243 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-credential-socket" pod "2a413a39-1efa-4c79-8a34-d40431400818" (UID: "2a413a39-1efa-4c79-8a34-d40431400818"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.132356 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.132320 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-envoy" pod "2a413a39-1efa-4c79-8a34-d40431400818" (UID: "2a413a39-1efa-4c79-8a34-d40431400818"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.133222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.132580 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a413a39-1efa-4c79-8a34-d40431400818-istio-podinfo" pod "2a413a39-1efa-4c79-8a34-d40431400818" (UID: "2a413a39-1efa-4c79-8a34-d40431400818"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:53:34.133222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.133058 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-kube-api-access-vqp26" pod "2a413a39-1efa-4c79-8a34-d40431400818" (UID: "2a413a39-1efa-4c79-8a34-d40431400818"). InnerVolumeSpecName "kube-api-access-vqp26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:34.133222 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.133076 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-istio-token" pod "2a413a39-1efa-4c79-8a34-d40431400818" (UID: "2a413a39-1efa-4c79-8a34-d40431400818"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:34.231294 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.231251 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a413a39-1efa-4c79-8a34-d40431400818-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.231294 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.231295 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.231464 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.231306 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.231464 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.231315 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.231464 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.231326 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a413a39-1efa-4c79-8a34-d40431400818-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.231464 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.231340 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqp26\" (UniqueName: \"kubernetes.io/projected/2a413a39-1efa-4c79-8a34-d40431400818-kube-api-access-vqp26\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.630634 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.630599 4304 generic.go:356] "Generic (PLEG): container finished" podID="2a413a39-1efa-4c79-8a34-d40431400818" containerID="ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997" exitCode=0 Nov 03 00:53:34.630865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.630696 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" Nov 03 00:53:34.630865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.630694 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" event={"ID":"2a413a39-1efa-4c79-8a34-d40431400818","Type":"ContainerDied","Data":"ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997"} Nov 03 00:53:34.630865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.630819 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc" event={"ID":"2a413a39-1efa-4c79-8a34-d40431400818","Type":"ContainerDied","Data":"482a1d42e835254e09c000c00559148ab6b2d7051161fe4dd8152e04f76d3542"} Nov 03 00:53:34.630865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.630850 4304 scope.go:122] "RemoveContainer" containerID="ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997" Nov 03 00:53:34.641810 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.641791 4304 scope.go:122] "RemoveContainer" containerID="ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997" Nov 03 00:53:34.642074 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:53:34.642052 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997\": container with ID starting with ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997 not found: ID does not exist" containerID="ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997" Nov 03 00:53:34.642164 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.642087 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997"} err="failed to get container status \"ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997\": rpc error: code = NotFound desc = could not find container \"ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997\": container with ID starting with ceec6b984acc24099f922a4bcd3f372c4a2f2e04d85decf3aa2d9963f12a2997 not found: ID does not exist" Nov 03 00:53:34.793647 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.793616 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc"] Nov 03 00:53:34.796966 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.796944 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hrbhc"] Nov 03 00:53:34.829417 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:53:34.829391 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a413a39-1efa-4c79-8a34-d40431400818" path="/var/lib/kubelet/pods/2a413a39-1efa-4c79-8a34-d40431400818/volumes" Nov 03 00:53:58.655906 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:53:58.655866 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:54:28.656520 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:54:28.656485 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:54:58.654531 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:54:58.654494 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:55:28.655191 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:55:28.655154 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:55:33.354207 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:33.354134 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn"] Nov 03 00:55:48.374252 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.374215 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" podUID="72d98d2f-4f35-49f9-9f4a-34986bb3799f" containerName="main" containerID="cri-o://9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad" gracePeriod=45 Nov 03 00:55:48.636855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.636798 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:55:48.712517 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.712486 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-home\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-home\") pod \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " Nov 03 00:55:48.712648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.712557 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tmp-dir\") pod \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " Nov 03 00:55:48.712648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.712620 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-model-cache\") pod \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " Nov 03 00:55:48.712756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.712714 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-home" pod "72d98d2f-4f35-49f9-9f4a-34986bb3799f" (UID: "72d98d2f-4f35-49f9-9f4a-34986bb3799f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:55:48.712756 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.712733 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d98d2f-4f35-49f9-9f4a-34986bb3799f-kube-api-access-4gv8z\" (UniqueName: \"kubernetes.io/projected/72d98d2f-4f35-49f9-9f4a-34986bb3799f-kube-api-access-4gv8z\") pod \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " Nov 03 00:55:48.712900 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.712842 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tls-certs\" (UniqueName: \"kubernetes.io/secret/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tls-certs\") pod \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " Nov 03 00:55:48.712900 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.712842 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tmp-dir" pod "72d98d2f-4f35-49f9-9f4a-34986bb3799f" (UID: "72d98d2f-4f35-49f9-9f4a-34986bb3799f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:55:48.712900 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.712881 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-model-cache" pod "72d98d2f-4f35-49f9-9f4a-34986bb3799f" (UID: "72d98d2f-4f35-49f9-9f4a-34986bb3799f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:55:48.712900 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.712898 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-dshm\") pod \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\" (UID: \"72d98d2f-4f35-49f9-9f4a-34986bb3799f\") " Nov 03 00:55:48.713285 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.713268 4304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-home\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:48.713370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.713292 4304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tmp-dir\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:48.713370 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.713304 4304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-model-cache\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:48.715028 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.715001 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-dshm" pod "72d98d2f-4f35-49f9-9f4a-34986bb3799f" (UID: "72d98d2f-4f35-49f9-9f4a-34986bb3799f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:55:48.715133 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.715055 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d98d2f-4f35-49f9-9f4a-34986bb3799f-kube-api-access-4gv8z" pod "72d98d2f-4f35-49f9-9f4a-34986bb3799f" (UID: "72d98d2f-4f35-49f9-9f4a-34986bb3799f"). InnerVolumeSpecName "kube-api-access-4gv8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:48.715133 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.715088 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tls-certs" pod "72d98d2f-4f35-49f9-9f4a-34986bb3799f" (UID: "72d98d2f-4f35-49f9-9f4a-34986bb3799f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:55:48.814607 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.814571 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gv8z\" (UniqueName: \"kubernetes.io/projected/72d98d2f-4f35-49f9-9f4a-34986bb3799f-kube-api-access-4gv8z\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:48.814607 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.814612 4304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d98d2f-4f35-49f9-9f4a-34986bb3799f-tls-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:48.814820 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:48.814625 4304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d98d2f-4f35-49f9-9f4a-34986bb3799f-dshm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:49.456364 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:49.456330 4304 generic.go:356] "Generic (PLEG): container finished" podID="72d98d2f-4f35-49f9-9f4a-34986bb3799f" containerID="9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad" exitCode=0 Nov 03 00:55:49.456841 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:49.456369 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" event={"ID":"72d98d2f-4f35-49f9-9f4a-34986bb3799f","Type":"ContainerDied","Data":"9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad"} Nov 03 00:55:49.456841 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:49.456370 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" Nov 03 00:55:49.456841 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:49.456403 4304 scope.go:122] "RemoveContainer" containerID="9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad" Nov 03 00:55:49.456841 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:49.456394 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" event={"ID":"72d98d2f-4f35-49f9-9f4a-34986bb3799f","Type":"ContainerDied","Data":"8f6d57c9ac3e12eeb98f321b9dcc684e535d461f2382c7e286bffc9dbaa9e2c3"} Nov 03 00:55:49.465653 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:49.465638 4304 scope.go:122] "RemoveContainer" containerID="9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad" Nov 03 00:55:49.465945 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:55:49.465922 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad\": container with ID starting with 9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad not found: ID does not exist" containerID="9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad" Nov 03 00:55:49.466024 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:49.465949 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad"} err="failed to get container status \"9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad\": rpc error: code = NotFound desc = could not find container \"9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad\": container with ID starting with 9948a4a6e7432af0a147da72782b9c2d0b328ea2c94d552ed031b90cf33788ad not found: ID does not exist" Nov 03 00:55:49.573644 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:49.573614 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn" podUID="72d98d2f-4f35-49f9-9f4a-34986bb3799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:49.620527 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:49.620498 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn"] Nov 03 00:55:49.670155 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:49.670132 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/auth-test-model-904a34-kserve-65f57b6cfc-8gqvn"] Nov 03 00:55:50.830052 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:50.830017 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d98d2f-4f35-49f9-9f4a-34986bb3799f" path="/var/lib/kubelet/pods/72d98d2f-4f35-49f9-9f4a-34986bb3799f/volumes" Nov 03 00:55:55.097075 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.097039 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp"] Nov 03 00:55:55.097692 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.097673 4304 cpu_manager.go:414] "containerMap: removing container" podUID="2a413a39-1efa-4c79-8a34-d40431400818" containerName="istio-proxy" Nov 03 00:55:55.097731 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.097695 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a413a39-1efa-4c79-8a34-d40431400818" containerName="istio-proxy" Nov 03 00:55:55.097731 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.097709 4304 cpu_manager.go:414] "containerMap: removing container" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerName="istio-proxy" Nov 03 00:55:55.097731 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.097714 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerName="istio-proxy" Nov 03 00:55:55.097932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.097733 4304 cpu_manager.go:414] "containerMap: removing container" podUID="72d98d2f-4f35-49f9-9f4a-34986bb3799f" containerName="main" Nov 03 00:55:55.097932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.097739 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d98d2f-4f35-49f9-9f4a-34986bb3799f" containerName="main" Nov 03 00:55:55.097932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.097840 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a413a39-1efa-4c79-8a34-d40431400818" containerName="istio-proxy" Nov 03 00:55:55.097932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.097850 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd16683-a7b5-4f26-8a6f-2a3160558426" containerName="istio-proxy" Nov 03 00:55:55.097932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.097858 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d98d2f-4f35-49f9-9f4a-34986bb3799f" containerName="main" Nov 03 00:55:55.102271 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.102255 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 00:55:55.206229 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.206190 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-115ee5-dockercfg-db22j\"" Nov 03 00:55:55.214190 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.214159 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-115ee5\"" Nov 03 00:55:55.218320 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.218293 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp"] Nov 03 00:55:55.258146 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.258115 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth"] Nov 03 00:55:55.262033 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.262017 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 00:55:55.283443 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.283417 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwp7l\" (UniqueName: \"kubernetes.io/projected/ef966559-e4ae-4ced-890d-2909a03807f7-kube-api-access-pwp7l\") pod \"payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp\" (UID: \"ef966559-e4ae-4ced-890d-2909a03807f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 00:55:55.283568 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.283528 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef966559-e4ae-4ced-890d-2909a03807f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp\" (UID: \"ef966559-e4ae-4ced-890d-2909a03807f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 00:55:55.312688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.312661 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth"] Nov 03 00:55:55.384981 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.384916 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwp7l\" (UniqueName: \"kubernetes.io/projected/ef966559-e4ae-4ced-890d-2909a03807f7-kube-api-access-pwp7l\") pod \"payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp\" (UID: \"ef966559-e4ae-4ced-890d-2909a03807f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 00:55:55.384981 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.384977 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef966559-e4ae-4ced-890d-2909a03807f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp\" (UID: \"ef966559-e4ae-4ced-890d-2909a03807f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 00:55:55.385196 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.384998 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08478de6-8d26-45ea-9310-b8daf70ab102-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth\" (UID: \"08478de6-8d26-45ea-9310-b8daf70ab102\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 00:55:55.385196 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.385038 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7vh\" (UniqueName: \"kubernetes.io/projected/08478de6-8d26-45ea-9310-b8daf70ab102-kube-api-access-pg7vh\") pod \"payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth\" (UID: \"08478de6-8d26-45ea-9310-b8daf70ab102\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 00:55:55.385716 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.385693 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef966559-e4ae-4ced-890d-2909a03807f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp\" (UID: \"ef966559-e4ae-4ced-890d-2909a03807f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 00:55:55.441275 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.441253 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwp7l\" (UniqueName: \"kubernetes.io/projected/ef966559-e4ae-4ced-890d-2909a03807f7-kube-api-access-pwp7l\") pod \"payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp\" (UID: \"ef966559-e4ae-4ced-890d-2909a03807f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 00:55:55.486019 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.485995 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08478de6-8d26-45ea-9310-b8daf70ab102-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth\" (UID: \"08478de6-8d26-45ea-9310-b8daf70ab102\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 00:55:55.486124 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.486042 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7vh\" (UniqueName: \"kubernetes.io/projected/08478de6-8d26-45ea-9310-b8daf70ab102-kube-api-access-pg7vh\") pod \"payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth\" (UID: \"08478de6-8d26-45ea-9310-b8daf70ab102\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 00:55:55.486649 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.486629 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08478de6-8d26-45ea-9310-b8daf70ab102-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth\" (UID: \"08478de6-8d26-45ea-9310-b8daf70ab102\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 00:55:55.542424 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.542393 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7vh\" (UniqueName: \"kubernetes.io/projected/08478de6-8d26-45ea-9310-b8daf70ab102-kube-api-access-pg7vh\") pod \"payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth\" (UID: \"08478de6-8d26-45ea-9310-b8daf70ab102\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 00:55:55.575434 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.575413 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 00:55:55.715402 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.715371 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 00:55:55.948152 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:55.948124 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth"] Nov 03 00:55:55.949568 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:55:55.949534 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08478de6_8d26_45ea_9310_b8daf70ab102.slice/crio-3c64dde91d6a11494a88c5ffed50b646f9f6675460231fd4e87ae862897f4a7c WatchSource:0}: Error finding container 3c64dde91d6a11494a88c5ffed50b646f9f6675460231fd4e87ae862897f4a7c: Status 404 returned error can't find the container with id 3c64dde91d6a11494a88c5ffed50b646f9f6675460231fd4e87ae862897f4a7c Nov 03 00:55:56.101363 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:56.101337 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp"] Nov 03 00:55:56.102589 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:55:56.102565 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef966559_e4ae_4ced_890d_2909a03807f7.slice/crio-04a6e1f07507c1701f9babaa026324ac0a17adafb3959c9cc2984e581e3aaae1 WatchSource:0}: Error finding container 04a6e1f07507c1701f9babaa026324ac0a17adafb3959c9cc2984e581e3aaae1: Status 404 returned error can't find the container with id 04a6e1f07507c1701f9babaa026324ac0a17adafb3959c9cc2984e581e3aaae1 Nov 03 00:55:56.505433 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:56.505395 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" event={"ID":"08478de6-8d26-45ea-9310-b8daf70ab102","Type":"ContainerStarted","Data":"fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59"} Nov 03 00:55:56.505616 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:56.505455 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" event={"ID":"08478de6-8d26-45ea-9310-b8daf70ab102","Type":"ContainerStarted","Data":"3c64dde91d6a11494a88c5ffed50b646f9f6675460231fd4e87ae862897f4a7c"} Nov 03 00:55:56.505694 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:56.505601 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 00:55:56.507039 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:56.507009 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" event={"ID":"ef966559-e4ae-4ced-890d-2909a03807f7","Type":"ContainerStarted","Data":"d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454"} Nov 03 00:55:56.507168 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:56.507045 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" event={"ID":"ef966559-e4ae-4ced-890d-2909a03807f7","Type":"ContainerStarted","Data":"04a6e1f07507c1701f9babaa026324ac0a17adafb3959c9cc2984e581e3aaae1"} Nov 03 00:55:56.507168 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:56.507107 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 00:55:56.619903 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:56.619855 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" podStartSLOduration=1.619841499 podStartE2EDuration="1.619841499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:56.617059153 +0000 UTC m=+3039.900616827" watchObservedRunningTime="2026-11-03 00:55:56.619841499 +0000 UTC m=+3039.903399171" Nov 03 00:55:56.729932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:55:56.729883 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" podStartSLOduration=1.7298658040000001 podStartE2EDuration="1.729865804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:56.728911526 +0000 UTC m=+3040.012469216" watchObservedRunningTime="2026-11-03 00:55:56.729865804 +0000 UTC m=+3040.013423478" Nov 03 00:55:58.655460 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:55:58.655424 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:56:07.515723 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:07.515695 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 00:56:07.516280 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:07.515759 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 00:56:24.158881 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:24.158841 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q"] Nov 03 00:56:28.656171 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:56:28.656134 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:56:39.179103 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.179049 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" podUID="06812e91-a171-4102-a6fa-bacb643db495" containerName="main" containerID="cri-o://42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a" gracePeriod=45 Nov 03 00:56:39.241141 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.241108 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" podUID="06812e91-a171-4102-a6fa-bacb643db495" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": dial tcp 10.133.0.138:8000: connect: connection refused" Nov 03 00:56:39.460141 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.460112 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:56:39.559594 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.559566 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-model-cache\") pod \"06812e91-a171-4102-a6fa-bacb643db495\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " Nov 03 00:56:39.559805 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.559655 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-tmp-dir\") pod \"06812e91-a171-4102-a6fa-bacb643db495\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " Nov 03 00:56:39.559805 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.559692 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06812e91-a171-4102-a6fa-bacb643db495-kube-api-access-fsd75\" (UniqueName: \"kubernetes.io/projected/06812e91-a171-4102-a6fa-bacb643db495-kube-api-access-fsd75\") pod \"06812e91-a171-4102-a6fa-bacb643db495\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " Nov 03 00:56:39.559805 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.559800 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06812e91-a171-4102-a6fa-bacb643db495-tls-certs\" (UniqueName: \"kubernetes.io/secret/06812e91-a171-4102-a6fa-bacb643db495-tls-certs\") pod \"06812e91-a171-4102-a6fa-bacb643db495\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " Nov 03 00:56:39.559976 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.559865 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-home\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-home\") pod \"06812e91-a171-4102-a6fa-bacb643db495\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " Nov 03 00:56:39.559976 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.559895 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-dshm\") pod \"06812e91-a171-4102-a6fa-bacb643db495\" (UID: \"06812e91-a171-4102-a6fa-bacb643db495\") " Nov 03 00:56:39.559976 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.559888 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-model-cache" pod "06812e91-a171-4102-a6fa-bacb643db495" (UID: "06812e91-a171-4102-a6fa-bacb643db495"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:56:39.559976 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.559958 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-tmp-dir" pod "06812e91-a171-4102-a6fa-bacb643db495" (UID: "06812e91-a171-4102-a6fa-bacb643db495"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:56:39.560179 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.560120 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-home" pod "06812e91-a171-4102-a6fa-bacb643db495" (UID: "06812e91-a171-4102-a6fa-bacb643db495"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:56:39.560727 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.560688 4304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-model-cache\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:39.560951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.560737 4304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-tmp-dir\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:39.560951 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.560752 4304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-home\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:39.562314 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.562263 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-dshm" pod "06812e91-a171-4102-a6fa-bacb643db495" (UID: "06812e91-a171-4102-a6fa-bacb643db495"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:56:39.562445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.562349 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06812e91-a171-4102-a6fa-bacb643db495-tls-certs" pod "06812e91-a171-4102-a6fa-bacb643db495" (UID: "06812e91-a171-4102-a6fa-bacb643db495"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:56:39.562445 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.562404 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06812e91-a171-4102-a6fa-bacb643db495-kube-api-access-fsd75" pod "06812e91-a171-4102-a6fa-bacb643db495" (UID: "06812e91-a171-4102-a6fa-bacb643db495"). InnerVolumeSpecName "kube-api-access-fsd75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:56:39.661968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.661938 4304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06812e91-a171-4102-a6fa-bacb643db495-dshm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:39.661968 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.661967 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsd75\" (UniqueName: \"kubernetes.io/projected/06812e91-a171-4102-a6fa-bacb643db495-kube-api-access-fsd75\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:39.662163 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.661981 4304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06812e91-a171-4102-a6fa-bacb643db495-tls-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:39.771103 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.771022 4304 generic.go:356] "Generic (PLEG): container finished" podID="06812e91-a171-4102-a6fa-bacb643db495" containerID="42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a" exitCode=0 Nov 03 00:56:39.771103 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.771070 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" Nov 03 00:56:39.771103 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.771078 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" event={"ID":"06812e91-a171-4102-a6fa-bacb643db495","Type":"ContainerDied","Data":"42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a"} Nov 03 00:56:39.771315 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.771116 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q" event={"ID":"06812e91-a171-4102-a6fa-bacb643db495","Type":"ContainerDied","Data":"3bfb91d38eb05d3410a0f4902cf149575aa5fc8a882887b5085f5fcc4b0a539c"} Nov 03 00:56:39.771315 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.771133 4304 scope.go:122] "RemoveContainer" containerID="42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a" Nov 03 00:56:39.781931 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.781914 4304 scope.go:122] "RemoveContainer" containerID="42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a" Nov 03 00:56:39.782185 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:56:39.782168 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a\": container with ID starting with 42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a not found: ID does not exist" containerID="42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a" Nov 03 00:56:39.782237 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.782192 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a"} err="failed to get container status \"42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a\": rpc error: code = NotFound desc = could not find container \"42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a\": container with ID starting with 42f8e37f2146152ad82d7702782c216962daed52b0663accf8f1125ab77bb70a not found: ID does not exist" Nov 03 00:56:39.934746 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.934717 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q"] Nov 03 00:56:39.937733 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:39.937710 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c8ce60/auth-test-model-c8ce60-kserve-75b7948bf-h7c6q"] Nov 03 00:56:40.426166 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:40.426136 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r"] Nov 03 00:56:40.829683 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:40.829609 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06812e91-a171-4102-a6fa-bacb643db495" path="/var/lib/kubelet/pods/06812e91-a171-4102-a6fa-bacb643db495/volumes" Nov 03 00:56:42.680153 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.680119 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq"] Nov 03 00:56:42.680652 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.680639 4304 cpu_manager.go:414] "containerMap: removing container" podUID="06812e91-a171-4102-a6fa-bacb643db495" containerName="main" Nov 03 00:56:42.680695 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.680653 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06812e91-a171-4102-a6fa-bacb643db495" containerName="main" Nov 03 00:56:42.680751 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.680742 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="06812e91-a171-4102-a6fa-bacb643db495" containerName="main" Nov 03 00:56:42.685326 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.685306 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.790306 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.790270 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq"] Nov 03 00:56:42.793875 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.793847 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-certs\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.794005 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.793903 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-socket\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.794005 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.793994 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-podinfo\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.794138 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.794122 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-credential-socket\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.794224 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.794209 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istiod-ca-cert\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.794277 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.794255 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-data\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.794332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.794296 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-token\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.794394 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.794374 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-envoy\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.794446 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.794399 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27d56\" (UniqueName: \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-kube-api-access-27d56\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.798270 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.798248 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c19567-openshift-default-dockercfg-7wjlg\"" Nov 03 00:56:42.895360 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.895326 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-certs\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.895552 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.895400 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-socket\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.895552 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.895430 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-podinfo\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.895688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.895576 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-credential-socket\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.895767 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.895738 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-certs\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.895876 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.895830 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-socket\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.895996 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.895968 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-credential-socket\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.896289 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.896066 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istiod-ca-cert\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.896289 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.896105 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-data\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.896289 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.896142 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-token\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.896289 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.896185 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-envoy\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.896521 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.896374 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27d56\" (UniqueName: \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-kube-api-access-27d56\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.896521 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.896424 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-data\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.896681 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.896661 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istiod-ca-cert\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.898179 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.898154 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-podinfo\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.898293 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.898276 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-envoy\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.953232 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.953166 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-token\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.953430 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.953413 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27d56\" (UniqueName: \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-kube-api-access-27d56\") pod \"e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:42.997795 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:42.997751 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:43.431085 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:43.431061 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq"] Nov 03 00:56:43.431570 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:56:43.431549 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399db5cb_e5e2_41f0_9e70_54958e3c8f70.slice/crio-997fe092670f2d2dbc8eb695df893bda524157656a801d936bbdc08d449db4fd WatchSource:0}: Error finding container 997fe092670f2d2dbc8eb695df893bda524157656a801d936bbdc08d449db4fd: Status 404 returned error can't find the container with id 997fe092670f2d2dbc8eb695df893bda524157656a801d936bbdc08d449db4fd Nov 03 00:56:43.434007 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:43.433972 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:56:43.434111 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:43.434050 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:56:43.434111 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:43.434094 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:56:43.800880 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:43.800803 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" event={"ID":"399db5cb-e5e2-41f0-9e70-54958e3c8f70","Type":"ContainerStarted","Data":"6921d8001a6b33274f62ae20dc3e6512256af5b677796767e5d4aaafdeb40629"} Nov 03 00:56:43.800880 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:43.800857 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" event={"ID":"399db5cb-e5e2-41f0-9e70-54958e3c8f70","Type":"ContainerStarted","Data":"997fe092670f2d2dbc8eb695df893bda524157656a801d936bbdc08d449db4fd"} Nov 03 00:56:43.921946 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:43.921880 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" podStartSLOduration=1.921863452 podStartE2EDuration="1.921863452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:43.918497192 +0000 UTC m=+3087.202054864" watchObservedRunningTime="2026-11-03 00:56:43.921863452 +0000 UTC m=+3087.205421126" Nov 03 00:56:43.998099 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:43.998069 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:45.004427 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:45.004396 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:45.813528 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:45.813494 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:45.814351 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:45.814329 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:56:55.446800 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.446735 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" podUID="b5339273-9767-488e-9064-bb7c35d6b4db" containerName="main" containerID="cri-o://4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9" gracePeriod=45 Nov 03 00:56:55.709962 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.709907 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:56:55.838304 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.838279 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5339273-9767-488e-9064-bb7c35d6b4db-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5339273-9767-488e-9064-bb7c35d6b4db-tls-certs\") pod \"b5339273-9767-488e-9064-bb7c35d6b4db\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " Nov 03 00:56:55.838452 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.838322 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-dshm\") pod \"b5339273-9767-488e-9064-bb7c35d6b4db\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " Nov 03 00:56:55.838452 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.838349 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5339273-9767-488e-9064-bb7c35d6b4db-kube-api-access-4d4jn\" (UniqueName: \"kubernetes.io/projected/b5339273-9767-488e-9064-bb7c35d6b4db-kube-api-access-4d4jn\") pod \"b5339273-9767-488e-9064-bb7c35d6b4db\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " Nov 03 00:56:55.838565 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.838543 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-home\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-home\") pod \"b5339273-9767-488e-9064-bb7c35d6b4db\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " Nov 03 00:56:55.838628 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.838594 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-model-cache\") pod \"b5339273-9767-488e-9064-bb7c35d6b4db\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " Nov 03 00:56:55.838976 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.838723 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-tmp-dir\") pod \"b5339273-9767-488e-9064-bb7c35d6b4db\" (UID: \"b5339273-9767-488e-9064-bb7c35d6b4db\") " Nov 03 00:56:55.838976 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.838863 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-home" pod "b5339273-9767-488e-9064-bb7c35d6b4db" (UID: "b5339273-9767-488e-9064-bb7c35d6b4db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:56:55.838976 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.838923 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-model-cache" pod "b5339273-9767-488e-9064-bb7c35d6b4db" (UID: "b5339273-9767-488e-9064-bb7c35d6b4db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:56:55.839229 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.839137 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-tmp-dir" pod "b5339273-9767-488e-9064-bb7c35d6b4db" (UID: "b5339273-9767-488e-9064-bb7c35d6b4db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:56:55.839817 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.839801 4304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-home\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:55.839881 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.839824 4304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-model-cache\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:55.839881 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.839837 4304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-tmp-dir\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:55.841244 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.841219 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5339273-9767-488e-9064-bb7c35d6b4db-tls-certs" pod "b5339273-9767-488e-9064-bb7c35d6b4db" (UID: "b5339273-9767-488e-9064-bb7c35d6b4db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:56:55.841358 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.841251 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5339273-9767-488e-9064-bb7c35d6b4db-kube-api-access-4d4jn" pod "b5339273-9767-488e-9064-bb7c35d6b4db" (UID: "b5339273-9767-488e-9064-bb7c35d6b4db"). InnerVolumeSpecName "kube-api-access-4d4jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:56:55.841358 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.841306 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-dshm" pod "b5339273-9767-488e-9064-bb7c35d6b4db" (UID: "b5339273-9767-488e-9064-bb7c35d6b4db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:56:55.881835 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.881812 4304 generic.go:356] "Generic (PLEG): container finished" podID="b5339273-9767-488e-9064-bb7c35d6b4db" containerID="4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9" exitCode=0 Nov 03 00:56:55.881932 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.881904 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" Nov 03 00:56:55.882009 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.881930 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" event={"ID":"b5339273-9767-488e-9064-bb7c35d6b4db","Type":"ContainerDied","Data":"4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9"} Nov 03 00:56:55.882009 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.881968 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r" event={"ID":"b5339273-9767-488e-9064-bb7c35d6b4db","Type":"ContainerDied","Data":"52a6be42afbef07ee9f8342d8f02aa55a48e483c20a5163b7a9c63d96affeee6"} Nov 03 00:56:55.882009 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.881987 4304 scope.go:122] "RemoveContainer" containerID="4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9" Nov 03 00:56:55.891597 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.891581 4304 scope.go:122] "RemoveContainer" containerID="4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9" Nov 03 00:56:55.891854 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:56:55.891839 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9\": container with ID starting with 4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9 not found: ID does not exist" containerID="4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9" Nov 03 00:56:55.891897 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.891860 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9"} err="failed to get container status \"4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9\": rpc error: code = NotFound desc = could not find container \"4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9\": container with ID starting with 4e96b20e6c314c59ce6abd9b259c468c8028b76974f55bd1602222feb544a7b9 not found: ID does not exist" Nov 03 00:56:55.941414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.941236 4304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5339273-9767-488e-9064-bb7c35d6b4db-dshm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:55.941414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.941264 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d4jn\" (UniqueName: \"kubernetes.io/projected/b5339273-9767-488e-9064-bb7c35d6b4db-kube-api-access-4d4jn\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:55.941414 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:55.941279 4304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5339273-9767-488e-9064-bb7c35d6b4db-tls-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:56.046198 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:56.046129 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r"] Nov 03 00:56:56.051953 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:56.051928 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-328e1b/auth-test-model-328e1b-kserve-66986df545-r5j9r"] Nov 03 00:56:56.830013 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:56:56.829981 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5339273-9767-488e-9064-bb7c35d6b4db" path="/var/lib/kubelet/pods/b5339273-9767-488e-9064-bb7c35d6b4db/volumes" Nov 03 00:56:58.655592 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:56:58.655558 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:57:02.176769 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.176693 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v"] Nov 03 00:57:02.177289 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.177271 4304 cpu_manager.go:414] "containerMap: removing container" podUID="b5339273-9767-488e-9064-bb7c35d6b4db" containerName="main" Nov 03 00:57:02.177340 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.177291 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5339273-9767-488e-9064-bb7c35d6b4db" containerName="main" Nov 03 00:57:02.177426 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.177416 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5339273-9767-488e-9064-bb7c35d6b4db" containerName="main" Nov 03 00:57:02.182122 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.182108 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.209563 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.209536 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-maas-api-tls\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.209688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.209573 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjzz8\" (UniqueName: \"kubernetes.io/projected/12d6acb4-ced4-4142-8c36-5d0821264540-kube-api-access-pjzz8\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.209732 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.209713 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d6acb4-ced4-4142-8c36-5d0821264540-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.209805 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.209790 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-metrics-certs\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.235315 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.235282 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-115ee5-serving-cert\"" Nov 03 00:57:02.287819 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.287788 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v"] Nov 03 00:57:02.310674 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.310636 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d6acb4-ced4-4142-8c36-5d0821264540-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.310822 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.310724 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-metrics-certs\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.310822 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.310807 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-maas-api-tls\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.310958 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.310853 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjzz8\" (UniqueName: \"kubernetes.io/projected/12d6acb4-ced4-4142-8c36-5d0821264540-kube-api-access-pjzz8\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.311228 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:57:02.311208 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-serving-cert: secret "maas-api-e2e-extmod-a-115ee5-serving-cert" not found Nov 03 00:57:02.311318 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:57:02.311295 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-maas-api-tls podName:12d6acb4-ced4-4142-8c36-5d0821264540 nodeName:}" failed. No retries permitted until 2026-11-03 00:57:02.811277352 +0000 UTC m=+3106.094835017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-maas-api-tls") pod "maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" (UID: "12d6acb4-ced4-4142-8c36-5d0821264540") : secret "maas-api-e2e-extmod-a-115ee5-serving-cert" not found Nov 03 00:57:02.311498 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.311454 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d6acb4-ced4-4142-8c36-5d0821264540-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.314104 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.314083 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-metrics-certs\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.367725 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.367699 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjzz8\" (UniqueName: \"kubernetes.io/projected/12d6acb4-ced4-4142-8c36-5d0821264540-kube-api-access-pjzz8\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.815818 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.815759 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-maas-api-tls\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:02.818187 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:02.818167 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-maas-api-tls\") pod \"maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:03.095568 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:03.095495 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:03.476080 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:03.476057 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v"] Nov 03 00:57:03.478006 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:57:03.477972 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d6acb4_ced4_4142_8c36_5d0821264540.slice/crio-a26c8e2daa81a3e0850d8e626478bbca99a30582175ee22a82a1c161ef787f59 WatchSource:0}: Error finding container a26c8e2daa81a3e0850d8e626478bbca99a30582175ee22a82a1c161ef787f59: Status 404 returned error can't find the container with id a26c8e2daa81a3e0850d8e626478bbca99a30582175ee22a82a1c161ef787f59 Nov 03 00:57:03.479555 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:03.479537 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:57:03.939557 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:03.939515 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" event={"ID":"12d6acb4-ced4-4142-8c36-5d0821264540","Type":"ContainerStarted","Data":"a26c8e2daa81a3e0850d8e626478bbca99a30582175ee22a82a1c161ef787f59"} Nov 03 00:57:04.947126 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:04.947087 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" event={"ID":"12d6acb4-ced4-4142-8c36-5d0821264540","Type":"ContainerStarted","Data":"a47298c285c21c5cfb4fc9d8d220b6b3960adc553db706295856fa62ea2a862f"} Nov 03 00:57:04.947553 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:04.947171 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:05.060723 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:05.060671 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" podStartSLOduration=2.223420662 podStartE2EDuration="3.060658075s" totalImagesPullingTime="837.237413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:05.058889159 +0000 UTC m=+3108.342446832" watchObservedRunningTime="2026-11-03 00:57:05.060658075 +0000 UTC m=+3108.344215747" Nov 03 00:57:10.960215 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:10.960186 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 00:57:28.655163 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:57:28.655126 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:57:50.509310 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.509277 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg"] Nov 03 00:57:50.513304 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.513285 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 00:57:50.616183 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.616156 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-3b1d36\"" Nov 03 00:57:50.616352 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.616187 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-3b1d36-dockercfg-jbq8z\"" Nov 03 00:57:50.619440 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.619416 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg"] Nov 03 00:57:50.633040 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.633015 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qflrm\" (UniqueName: \"kubernetes.io/projected/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-kube-api-access-qflrm\") pod \"payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg\" (UID: \"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 00:57:50.633228 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.633204 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg\" (UID: \"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 00:57:50.734250 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.734214 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qflrm\" (UniqueName: \"kubernetes.io/projected/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-kube-api-access-qflrm\") pod \"payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg\" (UID: \"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 00:57:50.734406 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.734354 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg\" (UID: \"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 00:57:50.735256 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.735229 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg\" (UID: \"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 00:57:50.791301 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.791242 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qflrm\" (UniqueName: \"kubernetes.io/projected/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-kube-api-access-qflrm\") pod \"payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg\" (UID: \"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 00:57:50.826829 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:50.826805 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 00:57:51.202404 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:51.202220 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg"] Nov 03 00:57:51.205276 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:57:51.205250 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f5aed5_6d09_4d18_9a20_0c4e31ad4731.slice/crio-9291a2b73d3ffd530a26025e2135b1da152d7be95843527daecff8a8ab3516a2 WatchSource:0}: Error finding container 9291a2b73d3ffd530a26025e2135b1da152d7be95843527daecff8a8ab3516a2: Status 404 returned error can't find the container with id 9291a2b73d3ffd530a26025e2135b1da152d7be95843527daecff8a8ab3516a2 Nov 03 00:57:51.281106 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:51.281071 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" event={"ID":"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731","Type":"ContainerStarted","Data":"9291a2b73d3ffd530a26025e2135b1da152d7be95843527daecff8a8ab3516a2"} Nov 03 00:57:52.289469 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:52.289422 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" event={"ID":"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731","Type":"ContainerStarted","Data":"7272e1366f2bae338e0c3bb9da0847000139358f1d531aa37a6432a596f2e183"} Nov 03 00:57:52.289883 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:52.289576 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 00:57:52.404212 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:57:52.404163 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" podStartSLOduration=2.404149377 podStartE2EDuration="2.404149377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:52.40296448 +0000 UTC m=+3155.686522153" watchObservedRunningTime="2026-11-03 00:57:52.404149377 +0000 UTC m=+3155.687707050" Nov 03 00:57:58.655131 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:57:58.655095 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:58:03.299117 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:03.299083 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 00:58:04.579440 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.579408 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6"] Nov 03 00:58:04.583485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.583468 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:04.638072 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.638052 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c19567-serving-cert\"" Nov 03 00:58:04.680799 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.680758 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vsdz\" (UniqueName: \"kubernetes.io/projected/0d7554aa-0be3-4d12-b48b-76937ff82345-kube-api-access-4vsdz\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:04.680938 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.680821 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-metrics-certs\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:04.680983 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.680955 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7554aa-0be3-4d12-b48b-76937ff82345-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:04.681026 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.681003 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:04.687391 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.687373 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6"] Nov 03 00:58:04.782569 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.782542 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7554aa-0be3-4d12-b48b-76937ff82345-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:04.782712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.782592 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:04.782712 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.782681 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsdz\" (UniqueName: \"kubernetes.io/projected/0d7554aa-0be3-4d12-b48b-76937ff82345-kube-api-access-4vsdz\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:04.782814 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.782716 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-metrics-certs\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:04.782814 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:58:04.782794 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-serving-cert: secret "maas-api-e2e-extmod-a-c19567-serving-cert" not found Nov 03 00:58:04.782890 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:58:04.782876 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-maas-api-tls podName:0d7554aa-0be3-4d12-b48b-76937ff82345 nodeName:}" failed. No retries permitted until 2026-11-03 00:58:05.28285293 +0000 UTC m=+3168.566410598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-maas-api-tls") pod "maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" (UID: "0d7554aa-0be3-4d12-b48b-76937ff82345") : secret "maas-api-e2e-extmod-a-c19567-serving-cert" not found Nov 03 00:58:04.783309 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.783290 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7554aa-0be3-4d12-b48b-76937ff82345-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:04.785139 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.785122 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-metrics-certs\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:04.840061 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:04.839985 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsdz\" (UniqueName: \"kubernetes.io/projected/0d7554aa-0be3-4d12-b48b-76937ff82345-kube-api-access-4vsdz\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:05.289178 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:05.289144 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:05.291632 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:05.291608 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:05.496441 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:05.496413 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:05.878760 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:05.878731 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6"] Nov 03 00:58:05.879944 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:58:05.879916 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7554aa_0be3_4d12_b48b_76937ff82345.slice/crio-5040a73503842ac4206da57f7428d8307e4e6f40f921da36c6cc16c6d6542cb7 WatchSource:0}: Error finding container 5040a73503842ac4206da57f7428d8307e4e6f40f921da36c6cc16c6d6542cb7: Status 404 returned error can't find the container with id 5040a73503842ac4206da57f7428d8307e4e6f40f921da36c6cc16c6d6542cb7 Nov 03 00:58:06.378196 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:06.378162 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" event={"ID":"0d7554aa-0be3-4d12-b48b-76937ff82345","Type":"ContainerStarted","Data":"5040a73503842ac4206da57f7428d8307e4e6f40f921da36c6cc16c6d6542cb7"} Nov 03 00:58:07.386688 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:07.386652 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" event={"ID":"0d7554aa-0be3-4d12-b48b-76937ff82345","Type":"ContainerStarted","Data":"b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2"} Nov 03 00:58:07.387159 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:07.386761 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:07.504254 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:07.504194 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" podStartSLOduration=2.590100183 podStartE2EDuration="3.504177434s" totalImagesPullingTime="914.077251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:07.499816106 +0000 UTC m=+3170.783373778" watchObservedRunningTime="2026-11-03 00:58:07.504177434 +0000 UTC m=+3170.787735107" Nov 03 00:58:13.398489 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:13.398461 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 00:58:13.995454 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:13.995418 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs"] Nov 03 00:58:13.999681 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:13.999658 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 00:58:14.052499 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:14.052473 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-087f6c\"" Nov 03 00:58:14.056741 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:14.056716 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs"] Nov 03 00:58:14.085564 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:14.085539 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a51c35-6437-4389-9847-ed53e39548cb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs\" (UID: \"47a51c35-6437-4389-9847-ed53e39548cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 00:58:14.085696 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:14.085581 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8j76\" (UniqueName: \"kubernetes.io/projected/47a51c35-6437-4389-9847-ed53e39548cb-kube-api-access-t8j76\") pod \"payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs\" (UID: \"47a51c35-6437-4389-9847-ed53e39548cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 00:58:14.101410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:14.101391 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-087f6c-dockercfg-zxqdr\"" Nov 03 00:58:14.186595 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:14.186567 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a51c35-6437-4389-9847-ed53e39548cb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs\" (UID: \"47a51c35-6437-4389-9847-ed53e39548cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 00:58:14.186691 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:14.186603 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8j76\" (UniqueName: \"kubernetes.io/projected/47a51c35-6437-4389-9847-ed53e39548cb-kube-api-access-t8j76\") pod \"payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs\" (UID: \"47a51c35-6437-4389-9847-ed53e39548cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 00:58:14.187332 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:14.187301 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a51c35-6437-4389-9847-ed53e39548cb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs\" (UID: \"47a51c35-6437-4389-9847-ed53e39548cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 00:58:14.244891 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:14.244862 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8j76\" (UniqueName: \"kubernetes.io/projected/47a51c35-6437-4389-9847-ed53e39548cb-kube-api-access-t8j76\") pod \"payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs\" (UID: \"47a51c35-6437-4389-9847-ed53e39548cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 00:58:14.313200 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:14.313141 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 00:58:14.688385 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:14.688364 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs"] Nov 03 00:58:14.690188 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:58:14.690163 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a51c35_6437_4389_9847_ed53e39548cb.slice/crio-ab9cd0e2fc7fddc45b6eb3dc0b7e5abfe8eea52d856d1c078dd658ac5c56e9c1 WatchSource:0}: Error finding container ab9cd0e2fc7fddc45b6eb3dc0b7e5abfe8eea52d856d1c078dd658ac5c56e9c1: Status 404 returned error can't find the container with id ab9cd0e2fc7fddc45b6eb3dc0b7e5abfe8eea52d856d1c078dd658ac5c56e9c1 Nov 03 00:58:15.440256 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:15.440217 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" event={"ID":"47a51c35-6437-4389-9847-ed53e39548cb","Type":"ContainerStarted","Data":"14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11"} Nov 03 00:58:15.440439 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:15.440284 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" event={"ID":"47a51c35-6437-4389-9847-ed53e39548cb","Type":"ContainerStarted","Data":"ab9cd0e2fc7fddc45b6eb3dc0b7e5abfe8eea52d856d1c078dd658ac5c56e9c1"} Nov 03 00:58:15.440439 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:15.440310 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 00:58:15.552927 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:15.552879 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" podStartSLOduration=2.552865847 podStartE2EDuration="2.552865847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:15.551858427 +0000 UTC m=+3178.835416099" watchObservedRunningTime="2026-11-03 00:58:15.552865847 +0000 UTC m=+3178.836423519" Nov 03 00:58:26.449855 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:26.449828 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 00:58:28.671815 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:58:28.671760 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:58:31.854945 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:31.854900 4304 patch_prober.go:29] interesting pod/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:58:31.855410 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:31.854967 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:34.585232 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:34.585162 4304 generic.go:356] "Generic (PLEG): container finished" podID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerID="6921d8001a6b33274f62ae20dc3e6512256af5b677796767e5d4aaafdeb40629" exitCode=0 Nov 03 00:58:34.585670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:34.585240 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" event={"ID":"399db5cb-e5e2-41f0-9e70-54958e3c8f70","Type":"ContainerDied","Data":"6921d8001a6b33274f62ae20dc3e6512256af5b677796767e5d4aaafdeb40629"} Nov 03 00:58:34.585670 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:34.585667 4304 scope.go:122] "RemoveContainer" containerID="6921d8001a6b33274f62ae20dc3e6512256af5b677796767e5d4aaafdeb40629" Nov 03 00:58:34.587543 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:34.587513 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:58:34.587619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:34.587603 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:58:34.587663 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:34.587649 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:58:35.593879 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:35.593834 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" event={"ID":"399db5cb-e5e2-41f0-9e70-54958e3c8f70","Type":"ContainerStarted","Data":"64b4d2663c8d950e4c9dee1e1a4328cac1c1ddbcdb94c52a90a738e567052617"} Nov 03 00:58:35.998464 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:35.998436 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:58:35.998648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:35.998546 4304 patch_prober.go:29] interesting pod/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 00:58:35.998648 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:35.998596 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 00:58:36.998415 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:36.998385 4304 patch_prober.go:29] interesting pod/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 00:58:36.998821 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:36.998435 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 00:58:37.998115 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:37.998081 4304 patch_prober.go:29] interesting pod/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 00:58:37.998284 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:37.998141 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 00:58:38.999069 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:38.999033 4304 patch_prober.go:29] interesting pod/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 00:58:38.999480 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:38.999119 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 00:58:39.998994 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:39.998953 4304 patch_prober.go:29] interesting pod/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 00:58:39.999204 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:39.999020 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 00:58:40.840599 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:40.840569 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t"] Nov 03 00:58:40.843585 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:40.843563 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 00:58:40.901892 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:40.901864 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t"] Nov 03 00:58:40.911922 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:40.911890 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrdjb\" (UniqueName: \"kubernetes.io/projected/ff68bdec-d726-476c-bdd7-8e0d902c2152-kube-api-access-jrdjb\") pod \"payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t\" (UID: \"ff68bdec-d726-476c-bdd7-8e0d902c2152\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 00:58:40.912200 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:40.912164 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff68bdec-d726-476c-bdd7-8e0d902c2152-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t\" (UID: \"ff68bdec-d726-476c-bdd7-8e0d902c2152\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 00:58:40.946684 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:40.946651 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-642e6a-dockercfg-qv48l\"" Nov 03 00:58:40.946684 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:40.946651 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-642e6a\"" Nov 03 00:58:41.002294 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.002270 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:58:41.002642 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.002547 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:58:41.003156 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.003141 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 00:58:41.012984 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.012959 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff68bdec-d726-476c-bdd7-8e0d902c2152-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t\" (UID: \"ff68bdec-d726-476c-bdd7-8e0d902c2152\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 00:58:41.013087 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.013058 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrdjb\" (UniqueName: \"kubernetes.io/projected/ff68bdec-d726-476c-bdd7-8e0d902c2152-kube-api-access-jrdjb\") pod \"payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t\" (UID: \"ff68bdec-d726-476c-bdd7-8e0d902c2152\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 00:58:41.013586 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.013569 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff68bdec-d726-476c-bdd7-8e0d902c2152-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t\" (UID: \"ff68bdec-d726-476c-bdd7-8e0d902c2152\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 00:58:41.069741 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.069715 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrdjb\" (UniqueName: \"kubernetes.io/projected/ff68bdec-d726-476c-bdd7-8e0d902c2152-kube-api-access-jrdjb\") pod \"payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t\" (UID: \"ff68bdec-d726-476c-bdd7-8e0d902c2152\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 00:58:41.156110 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.156083 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 00:58:41.542822 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.542801 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t"] Nov 03 00:58:41.544680 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:58:41.544654 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff68bdec_d726_476c_bdd7_8e0d902c2152.slice/crio-22eb60ce93191d7e8095e99030da10fe1207c59958a5c3b491536d054b072d20 WatchSource:0}: Error finding container 22eb60ce93191d7e8095e99030da10fe1207c59958a5c3b491536d054b072d20: Status 404 returned error can't find the container with id 22eb60ce93191d7e8095e99030da10fe1207c59958a5c3b491536d054b072d20 Nov 03 00:58:41.632757 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.632730 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" event={"ID":"ff68bdec-d726-476c-bdd7-8e0d902c2152","Type":"ContainerStarted","Data":"93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c"} Nov 03 00:58:41.632895 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.632797 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" event={"ID":"ff68bdec-d726-476c-bdd7-8e0d902c2152","Type":"ContainerStarted","Data":"22eb60ce93191d7e8095e99030da10fe1207c59958a5c3b491536d054b072d20"} Nov 03 00:58:41.633058 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.633031 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 00:58:41.749280 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:41.749236 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" podStartSLOduration=1.749223324 podStartE2EDuration="1.749223324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:41.746241448 +0000 UTC m=+3205.029799122" watchObservedRunningTime="2026-11-03 00:58:41.749223324 +0000 UTC m=+3205.032780996" Nov 03 00:58:52.643163 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:58:52.643126 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 00:58:58.654953 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:58:58.654902 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:59:17.235261 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.235223 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl"] Nov 03 00:59:17.240651 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.240631 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.286503 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.286478 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ebf98b-b935-44f9-8b9b-7ed9703fd340-openshift-service-ca\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.286619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.286512 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-maas-api-tls\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.286619 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.286536 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkx8\" (UniqueName: \"kubernetes.io/projected/12ebf98b-b935-44f9-8b9b-7ed9703fd340-kube-api-access-kgkx8\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.286761 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.286741 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-metrics-certs\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.296859 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.296834 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl"] Nov 03 00:59:17.341540 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.341514 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-ea50b7-serving-cert\"" Nov 03 00:59:17.387948 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.387919 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-metrics-certs\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.388093 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.388006 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ebf98b-b935-44f9-8b9b-7ed9703fd340-openshift-service-ca\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.388093 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.388040 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-maas-api-tls\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.388093 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.388063 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkx8\" (UniqueName: \"kubernetes.io/projected/12ebf98b-b935-44f9-8b9b-7ed9703fd340-kube-api-access-kgkx8\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.388373 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:59:17.388346 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-serving-cert: secret "maas-api-e2e-autores-ea50b7-serving-cert" not found Nov 03 00:59:17.388496 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:59:17.388440 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-maas-api-tls podName:12ebf98b-b935-44f9-8b9b-7ed9703fd340 nodeName:}" failed. No retries permitted until 2026-11-03 00:59:17.888417372 +0000 UTC m=+3241.171975029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-maas-api-tls") pod "maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" (UID: "12ebf98b-b935-44f9-8b9b-7ed9703fd340") : secret "maas-api-e2e-autores-ea50b7-serving-cert" not found Nov 03 00:59:17.388865 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.388844 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ebf98b-b935-44f9-8b9b-7ed9703fd340-openshift-service-ca\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.391110 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.391086 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-metrics-certs\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.444485 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.444457 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkx8\" (UniqueName: \"kubernetes.io/projected/12ebf98b-b935-44f9-8b9b-7ed9703fd340-kube-api-access-kgkx8\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.894455 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.894420 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-maas-api-tls\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:17.897006 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:17.896986 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-maas-api-tls\") pod \"maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:18.154098 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:18.154012 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:18.530955 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:18.530928 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl"] Nov 03 00:59:18.532291 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:59:18.532265 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ebf98b_b935_44f9_8b9b_7ed9703fd340.slice/crio-07044c6071303b25c0935a7b0d030b4aefdcc717948d2c2609c8e1515a04e14d WatchSource:0}: Error finding container 07044c6071303b25c0935a7b0d030b4aefdcc717948d2c2609c8e1515a04e14d: Status 404 returned error can't find the container with id 07044c6071303b25c0935a7b0d030b4aefdcc717948d2c2609c8e1515a04e14d Nov 03 00:59:18.869874 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:18.869769 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" event={"ID":"12ebf98b-b935-44f9-8b9b-7ed9703fd340","Type":"ContainerStarted","Data":"07044c6071303b25c0935a7b0d030b4aefdcc717948d2c2609c8e1515a04e14d"} Nov 03 00:59:19.877849 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:19.877802 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" event={"ID":"12ebf98b-b935-44f9-8b9b-7ed9703fd340","Type":"ContainerStarted","Data":"5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a"} Nov 03 00:59:19.878442 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:19.877883 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:19.993477 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:19.993425 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" podStartSLOduration=2.245440039 podStartE2EDuration="2.993410904s" totalImagesPullingTime="747.970865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:19.991277194 +0000 UTC m=+3243.274834866" watchObservedRunningTime="2026-11-03 00:59:19.993410904 +0000 UTC m=+3243.276968576" Nov 03 00:59:25.890941 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:25.890893 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 00:59:28.655283 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:59:28.655246 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 00:59:40.943451 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:40.943417 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987"] Nov 03 00:59:40.947607 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:40.947588 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.049538 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.049504 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbk89\" (UniqueName: \"kubernetes.io/projected/f405ee33-3d9a-4250-b334-0011a713a41f-kube-api-access-mbk89\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.049695 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.049610 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.049695 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.049637 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-metrics-certs\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.049836 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.049823 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-087f6c-serving-cert\"" Nov 03 00:59:41.049902 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.049848 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f405ee33-3d9a-4250-b334-0011a713a41f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.053449 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.053424 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987"] Nov 03 00:59:41.151359 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.151330 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f405ee33-3d9a-4250-b334-0011a713a41f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.151522 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.151409 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbk89\" (UniqueName: \"kubernetes.io/projected/f405ee33-3d9a-4250-b334-0011a713a41f-kube-api-access-mbk89\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.151522 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.151467 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.151522 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.151493 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-metrics-certs\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.151694 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:59:41.151605 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-serving-cert: secret "maas-api-e2e-extmod-a-087f6c-serving-cert" not found Nov 03 00:59:41.151804 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:59:41.151694 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-maas-api-tls podName:f405ee33-3d9a-4250-b334-0011a713a41f nodeName:}" failed. No retries permitted until 2026-11-03 00:59:41.651673394 +0000 UTC m=+3264.935231061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-maas-api-tls") pod "maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" (UID: "f405ee33-3d9a-4250-b334-0011a713a41f") : secret "maas-api-e2e-extmod-a-087f6c-serving-cert" not found Nov 03 00:59:41.152054 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.152034 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f405ee33-3d9a-4250-b334-0011a713a41f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.154219 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.154194 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-metrics-certs\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.208694 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.208623 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbk89\" (UniqueName: \"kubernetes.io/projected/f405ee33-3d9a-4250-b334-0011a713a41f-kube-api-access-mbk89\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.656638 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.656610 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.659136 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.659111 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-087f6c-58d775854d-wj987\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:41.862086 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:41.862064 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:42.260490 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:42.260464 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987"] Nov 03 00:59:42.261936 ip-10-0-140-172 kubenswrapper[4304]: W1103 00:59:42.261906 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf405ee33_3d9a_4250_b334_0011a713a41f.slice/crio-19c6dfa9577fb4bc83310a613108245b496af0a16b6f20ebd3a43b270a647571 WatchSource:0}: Error finding container 19c6dfa9577fb4bc83310a613108245b496af0a16b6f20ebd3a43b270a647571: Status 404 returned error can't find the container with id 19c6dfa9577fb4bc83310a613108245b496af0a16b6f20ebd3a43b270a647571 Nov 03 00:59:43.026480 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:43.026445 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" event={"ID":"f405ee33-3d9a-4250-b334-0011a713a41f","Type":"ContainerStarted","Data":"19c6dfa9577fb4bc83310a613108245b496af0a16b6f20ebd3a43b270a647571"} Nov 03 00:59:44.033969 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:44.033925 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" event={"ID":"f405ee33-3d9a-4250-b334-0011a713a41f","Type":"ContainerStarted","Data":"7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98"} Nov 03 00:59:44.034430 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:44.034000 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:44.161227 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:44.160708 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" podStartSLOduration=3.328797455 podStartE2EDuration="4.16068514s" totalImagesPullingTime="831.887685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:44.147541017 +0000 UTC m=+3267.431098710" watchObservedRunningTime="2026-11-03 00:59:44.16068514 +0000 UTC m=+3267.444242815" Nov 03 00:59:50.046237 ip-10-0-140-172 kubenswrapper[4304]: I1103 00:59:50.046211 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 00:59:58.655757 ip-10-0-140-172 kubenswrapper[4304]: E1103 00:59:58.655721 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:00:28.655762 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:00:28.655713 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:00:58.655157 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:00:58.655117 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:01:09.261421 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.261388 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl"] Nov 03 01:01:09.261937 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.261610 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" podUID="12ebf98b-b935-44f9-8b9b-7ed9703fd340" containerName="maas-api" containerID="cri-o://5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a" gracePeriod=30 Nov 03 01:01:09.547036 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.547004 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 01:01:09.610262 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.610232 4304 generic.go:356] "Generic (PLEG): container finished" podID="12ebf98b-b935-44f9-8b9b-7ed9703fd340" containerID="5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a" exitCode=0 Nov 03 01:01:09.610390 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.610274 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" event={"ID":"12ebf98b-b935-44f9-8b9b-7ed9703fd340","Type":"ContainerDied","Data":"5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a"} Nov 03 01:01:09.610390 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.610329 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" event={"ID":"12ebf98b-b935-44f9-8b9b-7ed9703fd340","Type":"ContainerDied","Data":"07044c6071303b25c0935a7b0d030b4aefdcc717948d2c2609c8e1515a04e14d"} Nov 03 01:01:09.610390 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.610350 4304 scope.go:122] "RemoveContainer" containerID="5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a" Nov 03 01:01:09.610390 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.610283 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl" Nov 03 01:01:09.619908 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.619890 4304 scope.go:122] "RemoveContainer" containerID="5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a" Nov 03 01:01:09.620177 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:01:09.620161 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a\": container with ID starting with 5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a not found: ID does not exist" containerID="5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a" Nov 03 01:01:09.620228 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.620185 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a"} err="failed to get container status \"5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a\": rpc error: code = NotFound desc = could not find container \"5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a\": container with ID starting with 5fc76c76198581b9292721cfdf4b90ec5ccc9fdd1b1b7182529a109fb4a2ac4a not found: ID does not exist" Nov 03 01:01:09.705888 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.705856 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-maas-api-tls\") pod \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " Nov 03 01:01:09.706087 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.706013 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ebf98b-b935-44f9-8b9b-7ed9703fd340-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ebf98b-b935-44f9-8b9b-7ed9703fd340-openshift-service-ca\") pod \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " Nov 03 01:01:09.706399 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.706374 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ebf98b-b935-44f9-8b9b-7ed9703fd340-openshift-service-ca" pod "12ebf98b-b935-44f9-8b9b-7ed9703fd340" (UID: "12ebf98b-b935-44f9-8b9b-7ed9703fd340"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:09.706753 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.706727 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-metrics-certs\") pod \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " Nov 03 01:01:09.706951 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.706796 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ebf98b-b935-44f9-8b9b-7ed9703fd340-kube-api-access-kgkx8\" (UniqueName: \"kubernetes.io/projected/12ebf98b-b935-44f9-8b9b-7ed9703fd340-kube-api-access-kgkx8\") pod \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\" (UID: \"12ebf98b-b935-44f9-8b9b-7ed9703fd340\") " Nov 03 01:01:09.707361 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.707331 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ebf98b-b935-44f9-8b9b-7ed9703fd340-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:09.709319 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.709292 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-maas-api-tls" pod "12ebf98b-b935-44f9-8b9b-7ed9703fd340" (UID: "12ebf98b-b935-44f9-8b9b-7ed9703fd340"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:01:09.710467 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.710289 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ebf98b-b935-44f9-8b9b-7ed9703fd340-kube-api-access-kgkx8" pod "12ebf98b-b935-44f9-8b9b-7ed9703fd340" (UID: "12ebf98b-b935-44f9-8b9b-7ed9703fd340"). InnerVolumeSpecName "kube-api-access-kgkx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:09.710599 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.710579 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-metrics-certs" pod "12ebf98b-b935-44f9-8b9b-7ed9703fd340" (UID: "12ebf98b-b935-44f9-8b9b-7ed9703fd340"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:01:09.808632 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.808552 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:09.808632 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.808585 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgkx8\" (UniqueName: \"kubernetes.io/projected/12ebf98b-b935-44f9-8b9b-7ed9703fd340-kube-api-access-kgkx8\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:09.808632 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:09.808596 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ebf98b-b935-44f9-8b9b-7ed9703fd340-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:10.029663 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:10.029609 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl"] Nov 03 01:01:10.033014 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:10.032992 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ea50b7-6d8d9c7b5b-dpjfl"] Nov 03 01:01:10.829719 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:10.829687 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ebf98b-b935-44f9-8b9b-7ed9703fd340" path="/var/lib/kubelet/pods/12ebf98b-b935-44f9-8b9b-7ed9703fd340/volumes" Nov 03 01:01:15.262421 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:15.262389 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs"] Nov 03 01:01:20.263598 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.263562 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" podUID="47a51c35-6437-4389-9847-ed53e39548cb" containerName="payload-processing" containerID="cri-o://14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11" gracePeriod=25 Nov 03 01:01:20.534119 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.534090 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 01:01:20.631585 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.631554 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a51c35-6437-4389-9847-ed53e39548cb-kube-api-access-t8j76\" (UniqueName: \"kubernetes.io/projected/47a51c35-6437-4389-9847-ed53e39548cb-kube-api-access-t8j76\") pod \"47a51c35-6437-4389-9847-ed53e39548cb\" (UID: \"47a51c35-6437-4389-9847-ed53e39548cb\") " Nov 03 01:01:20.631725 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.631665 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a51c35-6437-4389-9847-ed53e39548cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a51c35-6437-4389-9847-ed53e39548cb-plugins-config-volume\") pod \"47a51c35-6437-4389-9847-ed53e39548cb\" (UID: \"47a51c35-6437-4389-9847-ed53e39548cb\") " Nov 03 01:01:20.632039 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.632018 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a51c35-6437-4389-9847-ed53e39548cb-plugins-config-volume" pod "47a51c35-6437-4389-9847-ed53e39548cb" (UID: "47a51c35-6437-4389-9847-ed53e39548cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:20.632151 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.632138 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a51c35-6437-4389-9847-ed53e39548cb-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:20.633672 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.633650 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a51c35-6437-4389-9847-ed53e39548cb-kube-api-access-t8j76" pod "47a51c35-6437-4389-9847-ed53e39548cb" (UID: "47a51c35-6437-4389-9847-ed53e39548cb"). InnerVolumeSpecName "kube-api-access-t8j76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:20.682475 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.682444 4304 generic.go:356] "Generic (PLEG): container finished" podID="47a51c35-6437-4389-9847-ed53e39548cb" containerID="14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11" exitCode=0 Nov 03 01:01:20.682588 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.682492 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" Nov 03 01:01:20.682588 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.682525 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" event={"ID":"47a51c35-6437-4389-9847-ed53e39548cb","Type":"ContainerDied","Data":"14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11"} Nov 03 01:01:20.682588 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.682576 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs" event={"ID":"47a51c35-6437-4389-9847-ed53e39548cb","Type":"ContainerDied","Data":"ab9cd0e2fc7fddc45b6eb3dc0b7e5abfe8eea52d856d1c078dd658ac5c56e9c1"} Nov 03 01:01:20.682710 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.682596 4304 scope.go:122] "RemoveContainer" containerID="14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11" Nov 03 01:01:20.694134 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.694112 4304 scope.go:122] "RemoveContainer" containerID="14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11" Nov 03 01:01:20.694439 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:01:20.694418 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11\": container with ID starting with 14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11 not found: ID does not exist" containerID="14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11" Nov 03 01:01:20.694513 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.694447 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11"} err="failed to get container status \"14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11\": rpc error: code = NotFound desc = could not find container \"14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11\": container with ID starting with 14d3bc357d9742936ec8b0435eaeceb9a606614f79a6144f8b5f2afe033a4c11 not found: ID does not exist" Nov 03 01:01:20.732858 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.732837 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8j76\" (UniqueName: \"kubernetes.io/projected/47a51c35-6437-4389-9847-ed53e39548cb-kube-api-access-t8j76\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:20.846241 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.846170 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs"] Nov 03 01:01:20.849267 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:20.849247 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-087f6c-6cf5578f9-8hrhs"] Nov 03 01:01:22.829920 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:22.829886 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a51c35-6437-4389-9847-ed53e39548cb" path="/var/lib/kubelet/pods/47a51c35-6437-4389-9847-ed53e39548cb/volumes" Nov 03 01:01:25.899175 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:25.899147 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987"] Nov 03 01:01:25.899558 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:25.899328 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" podUID="f405ee33-3d9a-4250-b334-0011a713a41f" containerName="maas-api" containerID="cri-o://7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98" gracePeriod=30 Nov 03 01:01:26.171627 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.171569 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 01:01:26.289754 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.289729 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-metrics-certs\") pod \"f405ee33-3d9a-4250-b334-0011a713a41f\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " Nov 03 01:01:26.289921 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.289797 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f405ee33-3d9a-4250-b334-0011a713a41f-kube-api-access-mbk89\" (UniqueName: \"kubernetes.io/projected/f405ee33-3d9a-4250-b334-0011a713a41f-kube-api-access-mbk89\") pod \"f405ee33-3d9a-4250-b334-0011a713a41f\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " Nov 03 01:01:26.289921 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.289911 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-maas-api-tls\") pod \"f405ee33-3d9a-4250-b334-0011a713a41f\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " Nov 03 01:01:26.290012 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.289941 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f405ee33-3d9a-4250-b334-0011a713a41f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f405ee33-3d9a-4250-b334-0011a713a41f-openshift-service-ca\") pod \"f405ee33-3d9a-4250-b334-0011a713a41f\" (UID: \"f405ee33-3d9a-4250-b334-0011a713a41f\") " Nov 03 01:01:26.290315 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.290292 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f405ee33-3d9a-4250-b334-0011a713a41f-openshift-service-ca" pod "f405ee33-3d9a-4250-b334-0011a713a41f" (UID: "f405ee33-3d9a-4250-b334-0011a713a41f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:26.291924 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.291902 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-maas-api-tls" pod "f405ee33-3d9a-4250-b334-0011a713a41f" (UID: "f405ee33-3d9a-4250-b334-0011a713a41f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:01:26.292012 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.291971 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-metrics-certs" pod "f405ee33-3d9a-4250-b334-0011a713a41f" (UID: "f405ee33-3d9a-4250-b334-0011a713a41f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:01:26.292061 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.292040 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f405ee33-3d9a-4250-b334-0011a713a41f-kube-api-access-mbk89" pod "f405ee33-3d9a-4250-b334-0011a713a41f" (UID: "f405ee33-3d9a-4250-b334-0011a713a41f"). InnerVolumeSpecName "kube-api-access-mbk89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:26.391396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.391360 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:26.391396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.391389 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f405ee33-3d9a-4250-b334-0011a713a41f-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:26.391396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.391398 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f405ee33-3d9a-4250-b334-0011a713a41f-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:26.391677 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.391406 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbk89\" (UniqueName: \"kubernetes.io/projected/f405ee33-3d9a-4250-b334-0011a713a41f-kube-api-access-mbk89\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:26.721618 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.721584 4304 generic.go:356] "Generic (PLEG): container finished" podID="f405ee33-3d9a-4250-b334-0011a713a41f" containerID="7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98" exitCode=0 Nov 03 01:01:26.721806 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.721637 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" Nov 03 01:01:26.721806 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.721669 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" event={"ID":"f405ee33-3d9a-4250-b334-0011a713a41f","Type":"ContainerDied","Data":"7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98"} Nov 03 01:01:26.721806 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.721733 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987" event={"ID":"f405ee33-3d9a-4250-b334-0011a713a41f","Type":"ContainerDied","Data":"19c6dfa9577fb4bc83310a613108245b496af0a16b6f20ebd3a43b270a647571"} Nov 03 01:01:26.721806 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.721759 4304 scope.go:122] "RemoveContainer" containerID="7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98" Nov 03 01:01:26.732163 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.732146 4304 scope.go:122] "RemoveContainer" containerID="7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98" Nov 03 01:01:26.732442 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:01:26.732419 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98\": container with ID starting with 7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98 not found: ID does not exist" containerID="7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98" Nov 03 01:01:26.732505 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.732452 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98"} err="failed to get container status \"7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98\": rpc error: code = NotFound desc = could not find container \"7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98\": container with ID starting with 7aea59ef0cffc87a78c48f2f85f08536d226c21e15ab2dc7eb736f21ab3b5d98 not found: ID does not exist" Nov 03 01:01:26.885627 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.885599 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987"] Nov 03 01:01:26.888354 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:26.888332 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-087f6c-58d775854d-wj987"] Nov 03 01:01:28.655268 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:01:28.655232 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:01:28.829598 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:28.829567 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f405ee33-3d9a-4250-b334-0011a713a41f" path="/var/lib/kubelet/pods/f405ee33-3d9a-4250-b334-0011a713a41f/volumes" Nov 03 01:01:42.682155 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:42.682081 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v"] Nov 03 01:01:42.682669 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:42.682298 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" podUID="12d6acb4-ced4-4142-8c36-5d0821264540" containerName="maas-api" containerID="cri-o://a47298c285c21c5cfb4fc9d8d220b6b3960adc553db706295856fa62ea2a862f" gracePeriod=30 Nov 03 01:01:42.830841 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:42.830759 4304 generic.go:356] "Generic (PLEG): container finished" podID="12d6acb4-ced4-4142-8c36-5d0821264540" containerID="a47298c285c21c5cfb4fc9d8d220b6b3960adc553db706295856fa62ea2a862f" exitCode=0 Nov 03 01:01:42.834164 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:42.834125 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" event={"ID":"12d6acb4-ced4-4142-8c36-5d0821264540","Type":"ContainerDied","Data":"a47298c285c21c5cfb4fc9d8d220b6b3960adc553db706295856fa62ea2a862f"} Nov 03 01:01:42.964315 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:42.964288 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 01:01:43.006816 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.006765 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-metrics-certs\") pod \"12d6acb4-ced4-4142-8c36-5d0821264540\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " Nov 03 01:01:43.006939 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.006858 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d6acb4-ced4-4142-8c36-5d0821264540-kube-api-access-pjzz8\" (UniqueName: \"kubernetes.io/projected/12d6acb4-ced4-4142-8c36-5d0821264540-kube-api-access-pjzz8\") pod \"12d6acb4-ced4-4142-8c36-5d0821264540\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " Nov 03 01:01:43.006939 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.006932 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d6acb4-ced4-4142-8c36-5d0821264540-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d6acb4-ced4-4142-8c36-5d0821264540-openshift-service-ca\") pod \"12d6acb4-ced4-4142-8c36-5d0821264540\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " Nov 03 01:01:43.007052 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.007035 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-maas-api-tls\") pod \"12d6acb4-ced4-4142-8c36-5d0821264540\" (UID: \"12d6acb4-ced4-4142-8c36-5d0821264540\") " Nov 03 01:01:43.007305 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.007277 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d6acb4-ced4-4142-8c36-5d0821264540-openshift-service-ca" pod "12d6acb4-ced4-4142-8c36-5d0821264540" (UID: "12d6acb4-ced4-4142-8c36-5d0821264540"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:43.007579 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.007513 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d6acb4-ced4-4142-8c36-5d0821264540-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:43.009080 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.009055 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-metrics-certs" pod "12d6acb4-ced4-4142-8c36-5d0821264540" (UID: "12d6acb4-ced4-4142-8c36-5d0821264540"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:01:43.009080 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.009067 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d6acb4-ced4-4142-8c36-5d0821264540-kube-api-access-pjzz8" pod "12d6acb4-ced4-4142-8c36-5d0821264540" (UID: "12d6acb4-ced4-4142-8c36-5d0821264540"). InnerVolumeSpecName "kube-api-access-pjzz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:43.009216 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.009171 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-maas-api-tls" pod "12d6acb4-ced4-4142-8c36-5d0821264540" (UID: "12d6acb4-ced4-4142-8c36-5d0821264540"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:01:43.109846 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.108738 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:43.109846 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.108791 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d6acb4-ced4-4142-8c36-5d0821264540-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:43.109846 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.108808 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjzz8\" (UniqueName: \"kubernetes.io/projected/12d6acb4-ced4-4142-8c36-5d0821264540-kube-api-access-pjzz8\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:43.165487 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.165456 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth"] Nov 03 01:01:43.220381 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.220312 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp"] Nov 03 01:01:43.838998 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.838960 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" Nov 03 01:01:43.839489 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.839011 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v" event={"ID":"12d6acb4-ced4-4142-8c36-5d0821264540","Type":"ContainerDied","Data":"a26c8e2daa81a3e0850d8e626478bbca99a30582175ee22a82a1c161ef787f59"} Nov 03 01:01:43.839489 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:43.839070 4304 scope.go:122] "RemoveContainer" containerID="a47298c285c21c5cfb4fc9d8d220b6b3960adc553db706295856fa62ea2a862f" Nov 03 01:01:44.001717 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:44.001688 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v"] Nov 03 01:01:44.007644 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:44.007624 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-115ee5-5fd8855584-4262v"] Nov 03 01:01:44.830197 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:44.830164 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d6acb4-ced4-4142-8c36-5d0821264540" path="/var/lib/kubelet/pods/12d6acb4-ced4-4142-8c36-5d0821264540/volumes" Nov 03 01:01:48.166260 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.166217 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" podUID="08478de6-8d26-45ea-9310-b8daf70ab102" containerName="payload-processing" containerID="cri-o://fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59" gracePeriod=25 Nov 03 01:01:48.220636 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.220605 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" podUID="ef966559-e4ae-4ced-890d-2909a03807f7" containerName="payload-pre-processing" containerID="cri-o://d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454" gracePeriod=25 Nov 03 01:01:48.520671 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.520644 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 01:01:48.559220 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.559197 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 01:01:48.568666 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.568646 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08478de6-8d26-45ea-9310-b8daf70ab102-kube-api-access-pg7vh\" (UniqueName: \"kubernetes.io/projected/08478de6-8d26-45ea-9310-b8daf70ab102-kube-api-access-pg7vh\") pod \"08478de6-8d26-45ea-9310-b8daf70ab102\" (UID: \"08478de6-8d26-45ea-9310-b8daf70ab102\") " Nov 03 01:01:48.568764 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.568682 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08478de6-8d26-45ea-9310-b8daf70ab102-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08478de6-8d26-45ea-9310-b8daf70ab102-plugins-config-volume\") pod \"08478de6-8d26-45ea-9310-b8daf70ab102\" (UID: \"08478de6-8d26-45ea-9310-b8daf70ab102\") " Nov 03 01:01:48.569106 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.569082 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08478de6-8d26-45ea-9310-b8daf70ab102-plugins-config-volume" pod "08478de6-8d26-45ea-9310-b8daf70ab102" (UID: "08478de6-8d26-45ea-9310-b8daf70ab102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:48.570660 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.570640 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08478de6-8d26-45ea-9310-b8daf70ab102-kube-api-access-pg7vh" pod "08478de6-8d26-45ea-9310-b8daf70ab102" (UID: "08478de6-8d26-45ea-9310-b8daf70ab102"). InnerVolumeSpecName "kube-api-access-pg7vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:48.670301 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.670274 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef966559-e4ae-4ced-890d-2909a03807f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef966559-e4ae-4ced-890d-2909a03807f7-plugins-config-volume\") pod \"ef966559-e4ae-4ced-890d-2909a03807f7\" (UID: \"ef966559-e4ae-4ced-890d-2909a03807f7\") " Nov 03 01:01:48.670432 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.670383 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef966559-e4ae-4ced-890d-2909a03807f7-kube-api-access-pwp7l\" (UniqueName: \"kubernetes.io/projected/ef966559-e4ae-4ced-890d-2909a03807f7-kube-api-access-pwp7l\") pod \"ef966559-e4ae-4ced-890d-2909a03807f7\" (UID: \"ef966559-e4ae-4ced-890d-2909a03807f7\") " Nov 03 01:01:48.670728 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.670702 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef966559-e4ae-4ced-890d-2909a03807f7-plugins-config-volume" pod "ef966559-e4ae-4ced-890d-2909a03807f7" (UID: "ef966559-e4ae-4ced-890d-2909a03807f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:48.670888 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.670870 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg7vh\" (UniqueName: \"kubernetes.io/projected/08478de6-8d26-45ea-9310-b8daf70ab102-kube-api-access-pg7vh\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:48.670932 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.670895 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08478de6-8d26-45ea-9310-b8daf70ab102-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:48.670932 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.670909 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef966559-e4ae-4ced-890d-2909a03807f7-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:48.672576 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.672556 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef966559-e4ae-4ced-890d-2909a03807f7-kube-api-access-pwp7l" pod "ef966559-e4ae-4ced-890d-2909a03807f7" (UID: "ef966559-e4ae-4ced-890d-2909a03807f7"). InnerVolumeSpecName "kube-api-access-pwp7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:48.771947 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.771885 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwp7l\" (UniqueName: \"kubernetes.io/projected/ef966559-e4ae-4ced-890d-2909a03807f7-kube-api-access-pwp7l\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:48.877006 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.876979 4304 generic.go:356] "Generic (PLEG): container finished" podID="ef966559-e4ae-4ced-890d-2909a03807f7" containerID="d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454" exitCode=0 Nov 03 01:01:48.877107 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.877021 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" Nov 03 01:01:48.877107 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.877016 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" event={"ID":"ef966559-e4ae-4ced-890d-2909a03807f7","Type":"ContainerDied","Data":"d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454"} Nov 03 01:01:48.877198 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.877106 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp" event={"ID":"ef966559-e4ae-4ced-890d-2909a03807f7","Type":"ContainerDied","Data":"04a6e1f07507c1701f9babaa026324ac0a17adafb3959c9cc2984e581e3aaae1"} Nov 03 01:01:48.877198 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.877139 4304 scope.go:122] "RemoveContainer" containerID="d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454" Nov 03 01:01:48.878498 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.878473 4304 generic.go:356] "Generic (PLEG): container finished" podID="08478de6-8d26-45ea-9310-b8daf70ab102" containerID="fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59" exitCode=0 Nov 03 01:01:48.878592 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.878524 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" Nov 03 01:01:48.878592 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.878539 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" event={"ID":"08478de6-8d26-45ea-9310-b8daf70ab102","Type":"ContainerDied","Data":"fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59"} Nov 03 01:01:48.878592 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.878577 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth" event={"ID":"08478de6-8d26-45ea-9310-b8daf70ab102","Type":"ContainerDied","Data":"3c64dde91d6a11494a88c5ffed50b646f9f6675460231fd4e87ae862897f4a7c"} Nov 03 01:01:48.887082 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.887058 4304 scope.go:122] "RemoveContainer" containerID="d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454" Nov 03 01:01:48.887336 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:01:48.887320 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454\": container with ID starting with d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454 not found: ID does not exist" containerID="d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454" Nov 03 01:01:48.887383 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.887341 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454"} err="failed to get container status \"d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454\": rpc error: code = NotFound desc = could not find container \"d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454\": container with ID starting with d32d75a85d9c4d73812bf02fb274ca2862b15840a502d98a3ae2849c24097454 not found: ID does not exist" Nov 03 01:01:48.887383 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.887359 4304 scope.go:122] "RemoveContainer" containerID="fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59" Nov 03 01:01:48.896588 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.896570 4304 scope.go:122] "RemoveContainer" containerID="fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59" Nov 03 01:01:48.896851 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:01:48.896833 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59\": container with ID starting with fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59 not found: ID does not exist" containerID="fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59" Nov 03 01:01:48.896904 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:48.896857 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59"} err="failed to get container status \"fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59\": rpc error: code = NotFound desc = could not find container \"fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59\": container with ID starting with fe5e67b4508f2f3a7492f999443ff9d3c3c98a1861372ca4fe72e15373f30e59 not found: ID does not exist" Nov 03 01:01:49.039138 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:49.039082 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth"] Nov 03 01:01:49.042897 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:49.042879 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-115ee5-6db4597bc5-9jvth"] Nov 03 01:01:49.150143 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:49.150119 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp"] Nov 03 01:01:49.153037 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:49.153006 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-115ee5-69cc9cfb7f-glkkp"] Nov 03 01:01:50.830130 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:50.830100 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08478de6-8d26-45ea-9310-b8daf70ab102" path="/var/lib/kubelet/pods/08478de6-8d26-45ea-9310-b8daf70ab102/volumes" Nov 03 01:01:50.830500 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:01:50.830471 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef966559-e4ae-4ced-890d-2909a03807f7" path="/var/lib/kubelet/pods/ef966559-e4ae-4ced-890d-2909a03807f7/volumes" Nov 03 01:01:58.654683 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:01:58.654649 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:02:03.088495 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:03.088447 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t"] Nov 03 01:02:08.089537 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:08.089483 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" podUID="ff68bdec-d726-476c-bdd7-8e0d902c2152" containerName="payload-pre-processing" containerID="cri-o://93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c" gracePeriod=25 Nov 03 01:02:08.347647 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:08.347594 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 01:02:08.477341 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:08.477311 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff68bdec-d726-476c-bdd7-8e0d902c2152-kube-api-access-jrdjb\" (UniqueName: \"kubernetes.io/projected/ff68bdec-d726-476c-bdd7-8e0d902c2152-kube-api-access-jrdjb\") pod \"ff68bdec-d726-476c-bdd7-8e0d902c2152\" (UID: \"ff68bdec-d726-476c-bdd7-8e0d902c2152\") " Nov 03 01:02:08.477518 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:08.477424 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff68bdec-d726-476c-bdd7-8e0d902c2152-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff68bdec-d726-476c-bdd7-8e0d902c2152-plugins-config-volume\") pod \"ff68bdec-d726-476c-bdd7-8e0d902c2152\" (UID: \"ff68bdec-d726-476c-bdd7-8e0d902c2152\") " Nov 03 01:02:08.477895 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:08.477868 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff68bdec-d726-476c-bdd7-8e0d902c2152-plugins-config-volume" pod "ff68bdec-d726-476c-bdd7-8e0d902c2152" (UID: "ff68bdec-d726-476c-bdd7-8e0d902c2152"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:08.478069 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:08.478049 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff68bdec-d726-476c-bdd7-8e0d902c2152-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:08.479529 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:08.479507 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff68bdec-d726-476c-bdd7-8e0d902c2152-kube-api-access-jrdjb" pod "ff68bdec-d726-476c-bdd7-8e0d902c2152" (UID: "ff68bdec-d726-476c-bdd7-8e0d902c2152"). InnerVolumeSpecName "kube-api-access-jrdjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:08.579485 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:08.579461 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrdjb\" (UniqueName: \"kubernetes.io/projected/ff68bdec-d726-476c-bdd7-8e0d902c2152-kube-api-access-jrdjb\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:09.010396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:09.010364 4304 generic.go:356] "Generic (PLEG): container finished" podID="ff68bdec-d726-476c-bdd7-8e0d902c2152" containerID="93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c" exitCode=0 Nov 03 01:02:09.010565 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:09.010413 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" Nov 03 01:02:09.010565 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:09.010436 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" event={"ID":"ff68bdec-d726-476c-bdd7-8e0d902c2152","Type":"ContainerDied","Data":"93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c"} Nov 03 01:02:09.010565 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:09.010502 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t" event={"ID":"ff68bdec-d726-476c-bdd7-8e0d902c2152","Type":"ContainerDied","Data":"22eb60ce93191d7e8095e99030da10fe1207c59958a5c3b491536d054b072d20"} Nov 03 01:02:09.010565 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:09.010531 4304 scope.go:122] "RemoveContainer" containerID="93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c" Nov 03 01:02:09.020237 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:09.020220 4304 scope.go:122] "RemoveContainer" containerID="93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c" Nov 03 01:02:09.020499 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:02:09.020481 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c\": container with ID starting with 93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c not found: ID does not exist" containerID="93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c" Nov 03 01:02:09.020545 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:09.020508 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c"} err="failed to get container status \"93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c\": rpc error: code = NotFound desc = could not find container \"93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c\": container with ID starting with 93d331177d98a7696b9a7bcfa8500c96e1eedb59412428752e43f1e372b6f72c not found: ID does not exist" Nov 03 01:02:09.174184 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:09.174158 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t"] Nov 03 01:02:09.190155 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:09.190133 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-642e6a-6cc7dc584c-l9c4t"] Nov 03 01:02:10.831621 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:10.831589 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff68bdec-d726-476c-bdd7-8e0d902c2152" path="/var/lib/kubelet/pods/ff68bdec-d726-476c-bdd7-8e0d902c2152/volumes" Nov 03 01:02:24.951308 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:24.951279 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg"] Nov 03 01:02:28.655033 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:02:28.654999 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:02:29.951907 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:29.951870 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" podUID="e9f5aed5-6d09-4d18-9a20-0c4e31ad4731" containerName="payload-pre-processing" containerID="cri-o://7272e1366f2bae338e0c3bb9da0847000139358f1d531aa37a6432a596f2e183" gracePeriod=25 Nov 03 01:02:30.191974 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:30.191944 4304 generic.go:356] "Generic (PLEG): container finished" podID="e9f5aed5-6d09-4d18-9a20-0c4e31ad4731" containerID="7272e1366f2bae338e0c3bb9da0847000139358f1d531aa37a6432a596f2e183" exitCode=0 Nov 03 01:02:30.192187 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:30.192021 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" event={"ID":"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731","Type":"ContainerDied","Data":"7272e1366f2bae338e0c3bb9da0847000139358f1d531aa37a6432a596f2e183"} Nov 03 01:02:30.241872 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:30.241852 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 01:02:30.299407 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:30.299381 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-kube-api-access-qflrm\" (UniqueName: \"kubernetes.io/projected/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-kube-api-access-qflrm\") pod \"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731\" (UID: \"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731\") " Nov 03 01:02:30.299567 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:30.299458 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-plugins-config-volume\") pod \"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731\" (UID: \"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731\") " Nov 03 01:02:30.299811 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:30.299759 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-plugins-config-volume" pod "e9f5aed5-6d09-4d18-9a20-0c4e31ad4731" (UID: "e9f5aed5-6d09-4d18-9a20-0c4e31ad4731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:30.299903 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:30.299874 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:30.301536 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:30.301512 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-kube-api-access-qflrm" pod "e9f5aed5-6d09-4d18-9a20-0c4e31ad4731" (UID: "e9f5aed5-6d09-4d18-9a20-0c4e31ad4731"). InnerVolumeSpecName "kube-api-access-qflrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:30.400674 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:30.400648 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qflrm\" (UniqueName: \"kubernetes.io/projected/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731-kube-api-access-qflrm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:31.199877 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:31.199847 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" event={"ID":"e9f5aed5-6d09-4d18-9a20-0c4e31ad4731","Type":"ContainerDied","Data":"9291a2b73d3ffd530a26025e2135b1da152d7be95843527daecff8a8ab3516a2"} Nov 03 01:02:31.200334 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:31.199908 4304 scope.go:122] "RemoveContainer" containerID="7272e1366f2bae338e0c3bb9da0847000139358f1d531aa37a6432a596f2e183" Nov 03 01:02:31.200334 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:31.199907 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg" Nov 03 01:02:31.361878 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:31.361846 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg"] Nov 03 01:02:31.410836 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:31.410811 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3b1d36-6c7d75f679-n6pvg"] Nov 03 01:02:32.829717 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:32.829682 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f5aed5-6d09-4d18-9a20-0c4e31ad4731" path="/var/lib/kubelet/pods/e9f5aed5-6d09-4d18-9a20-0c4e31ad4731/volumes" Nov 03 01:02:45.318415 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.318385 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j"] Nov 03 01:02:45.318990 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.318973 4304 cpu_manager.go:414] "containerMap: removing container" podUID="ef966559-e4ae-4ced-890d-2909a03807f7" containerName="payload-pre-processing" Nov 03 01:02:45.319045 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.318992 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef966559-e4ae-4ced-890d-2909a03807f7" containerName="payload-pre-processing" Nov 03 01:02:45.319045 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319005 4304 cpu_manager.go:414] "containerMap: removing container" podUID="12ebf98b-b935-44f9-8b9b-7ed9703fd340" containerName="maas-api" Nov 03 01:02:45.319045 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319010 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ebf98b-b935-44f9-8b9b-7ed9703fd340" containerName="maas-api" Nov 03 01:02:45.319045 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319019 4304 cpu_manager.go:414] "containerMap: removing container" podUID="e9f5aed5-6d09-4d18-9a20-0c4e31ad4731" containerName="payload-pre-processing" Nov 03 01:02:45.319045 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319023 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f5aed5-6d09-4d18-9a20-0c4e31ad4731" containerName="payload-pre-processing" Nov 03 01:02:45.319045 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319039 4304 cpu_manager.go:414] "containerMap: removing container" podUID="ff68bdec-d726-476c-bdd7-8e0d902c2152" containerName="payload-pre-processing" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319046 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff68bdec-d726-476c-bdd7-8e0d902c2152" containerName="payload-pre-processing" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319058 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f405ee33-3d9a-4250-b334-0011a713a41f" containerName="maas-api" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319064 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f405ee33-3d9a-4250-b334-0011a713a41f" containerName="maas-api" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319078 4304 cpu_manager.go:414] "containerMap: removing container" podUID="12d6acb4-ced4-4142-8c36-5d0821264540" containerName="maas-api" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319083 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d6acb4-ced4-4142-8c36-5d0821264540" containerName="maas-api" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319096 4304 cpu_manager.go:414] "containerMap: removing container" podUID="08478de6-8d26-45ea-9310-b8daf70ab102" containerName="payload-processing" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319100 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08478de6-8d26-45ea-9310-b8daf70ab102" containerName="payload-processing" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319121 4304 cpu_manager.go:414] "containerMap: removing container" podUID="47a51c35-6437-4389-9847-ed53e39548cb" containerName="payload-processing" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319128 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a51c35-6437-4389-9847-ed53e39548cb" containerName="payload-processing" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319205 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff68bdec-d726-476c-bdd7-8e0d902c2152" containerName="payload-pre-processing" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319216 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f405ee33-3d9a-4250-b334-0011a713a41f" containerName="maas-api" Nov 03 01:02:45.319227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319225 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d6acb4-ced4-4142-8c36-5d0821264540" containerName="maas-api" Nov 03 01:02:45.319566 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319232 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="08478de6-8d26-45ea-9310-b8daf70ab102" containerName="payload-processing" Nov 03 01:02:45.319566 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319240 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a51c35-6437-4389-9847-ed53e39548cb" containerName="payload-processing" Nov 03 01:02:45.319566 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319246 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef966559-e4ae-4ced-890d-2909a03807f7" containerName="payload-pre-processing" Nov 03 01:02:45.319566 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319252 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ebf98b-b935-44f9-8b9b-7ed9703fd340" containerName="maas-api" Nov 03 01:02:45.319566 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.319258 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f5aed5-6d09-4d18-9a20-0c4e31ad4731" containerName="payload-pre-processing" Nov 03 01:02:45.323749 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.323732 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.375253 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.375228 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zx4cj\"" Nov 03 01:02:45.427906 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.427881 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j"] Nov 03 01:02:45.447851 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.447821 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.447994 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.447893 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mc8\" (UniqueName: \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-kube-api-access-m8mc8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.448069 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.448024 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.448128 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.448090 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.448206 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.448177 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.448264 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.448247 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.448304 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.448282 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.448354 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.448321 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.448418 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.448397 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550045 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550017 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mc8\" (UniqueName: \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-kube-api-access-m8mc8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550202 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550075 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550202 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550122 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550202 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550162 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550366 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550207 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550366 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550232 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550366 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550342 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550525 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550374 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550525 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550440 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550525 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550491 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550681 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550619 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550681 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550669 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.550759 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.550723 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.551350 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.551324 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.552720 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.552692 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.552826 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.552813 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.605857 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.605805 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mc8\" (UniqueName: \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-kube-api-access-m8mc8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.605973 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.605908 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:45.635950 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:45.635930 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:46.067993 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:46.067961 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j"] Nov 03 01:02:46.068563 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:02:46.068529 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e77ba6c_2a0e_4224_a790_ade61e6edf66.slice/crio-fc221a61ddfa171579733d47455f602b2610d911cd8b5dd69488cd02aa3a2b5e WatchSource:0}: Error finding container fc221a61ddfa171579733d47455f602b2610d911cd8b5dd69488cd02aa3a2b5e: Status 404 returned error can't find the container with id fc221a61ddfa171579733d47455f602b2610d911cd8b5dd69488cd02aa3a2b5e Nov 03 01:02:46.072179 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:46.072140 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:02:46.072352 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:46.072275 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:02:46.072416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:46.072357 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:02:46.297947 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:46.297908 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" event={"ID":"4e77ba6c-2a0e-4224-a790-ade61e6edf66","Type":"ContainerStarted","Data":"30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde"} Nov 03 01:02:46.298125 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:46.297993 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" event={"ID":"4e77ba6c-2a0e-4224-a790-ade61e6edf66","Type":"ContainerStarted","Data":"fc221a61ddfa171579733d47455f602b2610d911cd8b5dd69488cd02aa3a2b5e"} Nov 03 01:02:46.416641 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:46.416578 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" podStartSLOduration=1.416559135 podStartE2EDuration="1.416559135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:46.413496267 +0000 UTC m=+3449.697053940" watchObservedRunningTime="2026-11-03 01:02:46.416559135 +0000 UTC m=+3449.700116811" Nov 03 01:02:46.636739 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:46.636706 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:47.639009 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:47.638969 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 01:02:47.639603 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:47.639572 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 01:02:48.643799 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:48.643094 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 01:02:48.643799 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:48.643151 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 01:02:49.639143 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:49.639107 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 01:02:49.639566 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:49.639537 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 01:02:50.638799 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:50.638565 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 01:02:50.638799 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:50.638623 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 01:02:50.974355 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:50.974325 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6"] Nov 03 01:02:50.974765 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:50.974726 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" podUID="0d7554aa-0be3-4d12-b48b-76937ff82345" containerName="maas-api" containerID="cri-o://b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2" gracePeriod=30 Nov 03 01:02:51.290784 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.290750 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 01:02:51.352968 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.352932 4304 generic.go:356] "Generic (PLEG): container finished" podID="0d7554aa-0be3-4d12-b48b-76937ff82345" containerID="b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2" exitCode=0 Nov 03 01:02:51.353576 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.353197 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" event={"ID":"0d7554aa-0be3-4d12-b48b-76937ff82345","Type":"ContainerDied","Data":"b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2"} Nov 03 01:02:51.353797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.353763 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" event={"ID":"0d7554aa-0be3-4d12-b48b-76937ff82345","Type":"ContainerDied","Data":"5040a73503842ac4206da57f7428d8307e4e6f40f921da36c6cc16c6d6542cb7"} Nov 03 01:02:51.353936 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.353288 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6" Nov 03 01:02:51.354182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.354168 4304 scope.go:122] "RemoveContainer" containerID="b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2" Nov 03 01:02:51.370129 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.370100 4304 scope.go:122] "RemoveContainer" containerID="b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2" Nov 03 01:02:51.370667 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:02:51.370637 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2\": container with ID starting with b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2 not found: ID does not exist" containerID="b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2" Nov 03 01:02:51.370880 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.370680 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2"} err="failed to get container status \"b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2\": rpc error: code = NotFound desc = could not find container \"b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2\": container with ID starting with b694132573d60475ee197ffc8ccce892986358ec301dfe7986f5e0fb677271e2 not found: ID does not exist" Nov 03 01:02:51.417474 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.417450 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7554aa-0be3-4d12-b48b-76937ff82345-kube-api-access-4vsdz\" (UniqueName: \"kubernetes.io/projected/0d7554aa-0be3-4d12-b48b-76937ff82345-kube-api-access-4vsdz\") pod \"0d7554aa-0be3-4d12-b48b-76937ff82345\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " Nov 03 01:02:51.417682 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.417669 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-maas-api-tls\") pod \"0d7554aa-0be3-4d12-b48b-76937ff82345\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " Nov 03 01:02:51.417811 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.417794 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-metrics-certs\") pod \"0d7554aa-0be3-4d12-b48b-76937ff82345\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " Nov 03 01:02:51.418028 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.418009 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7554aa-0be3-4d12-b48b-76937ff82345-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7554aa-0be3-4d12-b48b-76937ff82345-openshift-service-ca\") pod \"0d7554aa-0be3-4d12-b48b-76937ff82345\" (UID: \"0d7554aa-0be3-4d12-b48b-76937ff82345\") " Nov 03 01:02:51.418951 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.418934 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7554aa-0be3-4d12-b48b-76937ff82345-openshift-service-ca" pod "0d7554aa-0be3-4d12-b48b-76937ff82345" (UID: "0d7554aa-0be3-4d12-b48b-76937ff82345"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:51.454005 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.453968 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-metrics-certs" pod "0d7554aa-0be3-4d12-b48b-76937ff82345" (UID: "0d7554aa-0be3-4d12-b48b-76937ff82345"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:51.454418 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.454394 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7554aa-0be3-4d12-b48b-76937ff82345-kube-api-access-4vsdz" pod "0d7554aa-0be3-4d12-b48b-76937ff82345" (UID: "0d7554aa-0be3-4d12-b48b-76937ff82345"). InnerVolumeSpecName "kube-api-access-4vsdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:51.456184 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.456167 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-maas-api-tls" pod "0d7554aa-0be3-4d12-b48b-76937ff82345" (UID: "0d7554aa-0be3-4d12-b48b-76937ff82345"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:51.519922 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.519848 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7554aa-0be3-4d12-b48b-76937ff82345-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:51.519922 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.519875 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vsdz\" (UniqueName: \"kubernetes.io/projected/0d7554aa-0be3-4d12-b48b-76937ff82345-kube-api-access-4vsdz\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:51.519922 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.519892 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:51.519922 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.519906 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7554aa-0be3-4d12-b48b-76937ff82345-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:51.642800 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.638160 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 01:02:51.642800 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.638221 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 01:02:51.773480 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.773400 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6"] Nov 03 01:02:51.777103 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:51.777078 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19567-7c99b7cd95-vrfp6"] Nov 03 01:02:52.637203 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:52.637171 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 01:02:52.637380 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:52.637231 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 01:02:52.830559 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:52.830525 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d7554aa-0be3-4d12-b48b-76937ff82345" path="/var/lib/kubelet/pods/0d7554aa-0be3-4d12-b48b-76937ff82345/volumes" Nov 03 01:02:53.247262 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:53.247228 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq"] Nov 03 01:02:53.247471 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:53.247438 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" containerID="cri-o://64b4d2663c8d950e4c9dee1e1a4328cac1c1ddbcdb94c52a90a738e567052617" gracePeriod=30 Nov 03 01:02:53.637150 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:53.637066 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 01:02:53.637150 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:53.637123 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 01:02:54.942846 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:54.942813 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:54.943382 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:54.943361 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:54.945639 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:54.945617 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:02:56.004496 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:56.004462 4304 patch_prober.go:29] interesting pod/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:02:56.004892 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:56.004521 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:02:58.414399 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.414369 4304 generic.go:356] "Generic (PLEG): container finished" podID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerID="64b4d2663c8d950e4c9dee1e1a4328cac1c1ddbcdb94c52a90a738e567052617" exitCode=0 Nov 03 01:02:58.414792 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.414446 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" event={"ID":"399db5cb-e5e2-41f0-9e70-54958e3c8f70","Type":"ContainerDied","Data":"64b4d2663c8d950e4c9dee1e1a4328cac1c1ddbcdb94c52a90a738e567052617"} Nov 03 01:02:58.414792 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.414504 4304 scope.go:122] "RemoveContainer" containerID="6921d8001a6b33274f62ae20dc3e6512256af5b677796767e5d4aaafdeb40629" Nov 03 01:02:58.533628 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.533597 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 01:02:58.627045 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627016 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-socket\") pod \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " Nov 03 01:02:58.627225 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627050 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istiod-ca-cert\") pod \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " Nov 03 01:02:58.627225 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627101 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-certs\") pod \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " Nov 03 01:02:58.627225 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627132 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-podinfo\") pod \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " Nov 03 01:02:58.627225 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627218 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-envoy\") pod \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " Nov 03 01:02:58.627464 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627254 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-data\") pod \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " Nov 03 01:02:58.627464 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627320 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-socket" pod "399db5cb-e5e2-41f0-9e70-54958e3c8f70" (UID: "399db5cb-e5e2-41f0-9e70-54958e3c8f70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:58.627464 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627426 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-certs" pod "399db5cb-e5e2-41f0-9e70-54958e3c8f70" (UID: "399db5cb-e5e2-41f0-9e70-54958e3c8f70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:58.627636 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627487 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-data" pod "399db5cb-e5e2-41f0-9e70-54958e3c8f70" (UID: "399db5cb-e5e2-41f0-9e70-54958e3c8f70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:58.627636 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627521 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-credential-socket\") pod \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " Nov 03 01:02:58.627636 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627575 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-kube-api-access-27d56\" (UniqueName: \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-kube-api-access-27d56\") pod \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " Nov 03 01:02:58.627636 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627609 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-token\" (UniqueName: \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-token\") pod \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\" (UID: \"399db5cb-e5e2-41f0-9e70-54958e3c8f70\") " Nov 03 01:02:58.627895 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.627810 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-credential-socket" pod "399db5cb-e5e2-41f0-9e70-54958e3c8f70" (UID: "399db5cb-e5e2-41f0-9e70-54958e3c8f70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:58.628124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.628047 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:58.628124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.628072 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:58.628124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.628086 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:58.628124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.628100 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:58.628449 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.628432 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istiod-ca-cert" pod "399db5cb-e5e2-41f0-9e70-54958e3c8f70" (UID: "399db5cb-e5e2-41f0-9e70-54958e3c8f70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:58.630239 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.630212 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-envoy" pod "399db5cb-e5e2-41f0-9e70-54958e3c8f70" (UID: "399db5cb-e5e2-41f0-9e70-54958e3c8f70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:58.631185 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.631159 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-token" pod "399db5cb-e5e2-41f0-9e70-54958e3c8f70" (UID: "399db5cb-e5e2-41f0-9e70-54958e3c8f70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:58.631525 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.631499 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-kube-api-access-27d56" pod "399db5cb-e5e2-41f0-9e70-54958e3c8f70" (UID: "399db5cb-e5e2-41f0-9e70-54958e3c8f70"). InnerVolumeSpecName "kube-api-access-27d56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:58.632037 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.631977 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-podinfo" pod "399db5cb-e5e2-41f0-9e70-54958e3c8f70" (UID: "399db5cb-e5e2-41f0-9e70-54958e3c8f70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:02:58.657798 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:02:58.657748 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:02:58.729903 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.729866 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:58.729903 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.729903 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27d56\" (UniqueName: \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-kube-api-access-27d56\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:58.730060 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.729921 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:58.730060 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.729936 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:58.730060 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:58.729950 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399db5cb-e5e2-41f0-9e70-54958e3c8f70-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:59.422316 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:59.422250 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" Nov 03 01:02:59.422715 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:59.422248 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq" event={"ID":"399db5cb-e5e2-41f0-9e70-54958e3c8f70","Type":"ContainerDied","Data":"997fe092670f2d2dbc8eb695df893bda524157656a801d936bbdc08d449db4fd"} Nov 03 01:02:59.422715 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:59.422397 4304 scope.go:122] "RemoveContainer" containerID="64b4d2663c8d950e4c9dee1e1a4328cac1c1ddbcdb94c52a90a738e567052617" Nov 03 01:02:59.636560 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:59.636532 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq"] Nov 03 01:02:59.640625 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:02:59.640598 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c19567-openshift-default-75cf96f465-bdzrq"] Nov 03 01:03:00.829536 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:00.829491 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" path="/var/lib/kubelet/pods/399db5cb-e5e2-41f0-9e70-54958e3c8f70/volumes" Nov 03 01:03:28.658410 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:03:28.658375 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:03:39.501596 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.501560 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8"] Nov 03 01:03:39.503946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.502249 4304 cpu_manager.go:414] "containerMap: removing container" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" Nov 03 01:03:39.503946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.502266 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" Nov 03 01:03:39.503946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.502288 4304 cpu_manager.go:414] "containerMap: removing container" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" Nov 03 01:03:39.503946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.502295 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" Nov 03 01:03:39.503946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.502314 4304 cpu_manager.go:414] "containerMap: removing container" podUID="0d7554aa-0be3-4d12-b48b-76937ff82345" containerName="maas-api" Nov 03 01:03:39.503946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.502321 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7554aa-0be3-4d12-b48b-76937ff82345" containerName="maas-api" Nov 03 01:03:39.503946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.502425 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" Nov 03 01:03:39.503946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.502433 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="399db5cb-e5e2-41f0-9e70-54958e3c8f70" containerName="istio-proxy" Nov 03 01:03:39.503946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.502441 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7554aa-0be3-4d12-b48b-76937ff82345" containerName="maas-api" Nov 03 01:03:39.505394 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.505376 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.562838 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.562800 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-token\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.562970 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.562840 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-data\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.563020 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.562967 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-socket\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.563056 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.563029 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70c8d9a-48b8-48f2-b985-001e2448f741-istiod-ca-cert\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.563102 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.563084 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-envoy\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.563278 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.563252 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-credential-socket\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.563383 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.563365 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b7sr\" (UniqueName: \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-kube-api-access-6b7sr\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.563460 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.563436 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-podinfo\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.563522 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.563505 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-certs\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.608720 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.608695 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8"] Nov 03 01:03:39.654839 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.654769 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4db6f5-openshift-default-dockercfg-xxbxl\"" Nov 03 01:03:39.676579 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.676528 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-socket\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.676806 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.676603 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70c8d9a-48b8-48f2-b985-001e2448f741-istiod-ca-cert\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.676806 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.676657 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-envoy\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.676806 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.676703 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-credential-socket\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.676806 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.676802 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7sr\" (UniqueName: \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-kube-api-access-6b7sr\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.677060 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.676836 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-podinfo\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.677060 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.676875 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-certs\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.677060 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.676954 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-token\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.677060 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.676981 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-data\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.690479 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.690449 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-credential-socket\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.691062 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.691040 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-socket\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.691978 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.691807 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70c8d9a-48b8-48f2-b985-001e2448f741-istiod-ca-cert\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.709239 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.703382 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-certs\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.709239 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.703881 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-envoy\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.709239 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.707849 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-data\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.720827 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.720796 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-podinfo\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.759636 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.759579 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b7sr\" (UniqueName: \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-kube-api-access-6b7sr\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.759857 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.759842 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-token\") pod \"e2e-inf-a-4db6f5-openshift-default-888874649-whqt8\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:39.817554 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:39.817520 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:40.244854 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:40.244828 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8"] Nov 03 01:03:40.245722 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:03:40.245697 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70c8d9a_48b8_48f2_b985_001e2448f741.slice/crio-dcd218bcc9048bdbe2223fbf5fbb876526428ab7fea134153f010c9580f8d42d WatchSource:0}: Error finding container dcd218bcc9048bdbe2223fbf5fbb876526428ab7fea134153f010c9580f8d42d: Status 404 returned error can't find the container with id dcd218bcc9048bdbe2223fbf5fbb876526428ab7fea134153f010c9580f8d42d Nov 03 01:03:40.248197 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:40.248164 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:40.248290 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:40.248244 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:40.248369 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:40.248291 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:40.731199 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:40.731159 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" event={"ID":"e70c8d9a-48b8-48f2-b985-001e2448f741","Type":"ContainerStarted","Data":"866f977aa251107cea02af898260fc7dd824cc29c85825b97d984c955832fca1"} Nov 03 01:03:40.731579 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:40.731246 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" event={"ID":"e70c8d9a-48b8-48f2-b985-001e2448f741","Type":"ContainerStarted","Data":"dcd218bcc9048bdbe2223fbf5fbb876526428ab7fea134153f010c9580f8d42d"} Nov 03 01:03:40.817819 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:40.817792 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:40.849274 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:40.849224 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" podStartSLOduration=1.849210995 podStartE2EDuration="1.849210995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:40.845815482 +0000 UTC m=+3504.129373154" watchObservedRunningTime="2026-11-03 01:03:40.849210995 +0000 UTC m=+3504.132768715" Nov 03 01:03:41.824108 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:41.824077 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:42.743923 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:42.743891 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:42.744626 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:42.744606 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:03:45.777613 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.777583 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2"] Nov 03 01:03:45.781549 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.781533 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.845749 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.845725 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.845908 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.845760 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnl9x\" (UniqueName: \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-kube-api-access-cnl9x\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.845908 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.845880 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.845985 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.845910 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.846023 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.846002 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.846068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.846052 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.846103 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.846077 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.846159 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.846148 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262aac2d-8690-4618-89f9-32c9e2eb8226-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.846194 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.846180 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947150 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947119 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262aac2d-8690-4618-89f9-32c9e2eb8226-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947150 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947158 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947418 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947230 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947418 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947259 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnl9x\" (UniqueName: \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-kube-api-access-cnl9x\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947418 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947316 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947418 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947332 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947647 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947410 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947788 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947749 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947874 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947826 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947874 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947849 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947993 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947904 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.947993 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.947942 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262aac2d-8690-4618-89f9-32c9e2eb8226-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.948105 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.948087 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.948187 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.948165 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.950142 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.950118 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.950308 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.950290 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:45.988635 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:45.988604 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2"] Nov 03 01:03:46.003503 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:46.003480 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:46.003799 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:46.003759 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnl9x\" (UniqueName: \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-kube-api-access-cnl9x\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:46.095004 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:46.094937 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:46.523483 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:46.523457 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2"] Nov 03 01:03:46.525417 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:03:46.525386 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262aac2d_8690_4618_89f9_32c9e2eb8226.slice/crio-f9473afa167738f95f45d906864e5c3a3f0e96192835dbe6fd92597a95b79a42 WatchSource:0}: Error finding container f9473afa167738f95f45d906864e5c3a3f0e96192835dbe6fd92597a95b79a42: Status 404 returned error can't find the container with id f9473afa167738f95f45d906864e5c3a3f0e96192835dbe6fd92597a95b79a42 Nov 03 01:03:46.528106 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:46.528076 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:46.528192 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:46.528141 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:46.528192 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:46.528169 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:46.770503 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:46.770460 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" event={"ID":"262aac2d-8690-4618-89f9-32c9e2eb8226","Type":"ContainerStarted","Data":"b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc"} Nov 03 01:03:46.770654 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:46.770519 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" event={"ID":"262aac2d-8690-4618-89f9-32c9e2eb8226","Type":"ContainerStarted","Data":"f9473afa167738f95f45d906864e5c3a3f0e96192835dbe6fd92597a95b79a42"} Nov 03 01:03:46.888364 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:46.888260 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" podStartSLOduration=1.8882429950000001 podStartE2EDuration="1.888242995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:46.885290824 +0000 UTC m=+3510.168848496" watchObservedRunningTime="2026-11-03 01:03:46.888242995 +0000 UTC m=+3510.171800670" Nov 03 01:03:47.095446 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:47.095414 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:47.096903 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:47.096879 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:47.097048 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:47.096931 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 01:03:48.095520 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:48.095486 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:48.095940 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:48.095535 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 01:03:49.095268 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:49.095238 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:49.095443 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:49.095287 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 01:03:50.095198 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:50.095162 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:50.095596 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:50.095219 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 01:03:51.095396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:51.095361 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:51.095808 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:51.095411 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 01:03:52.095297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:52.095266 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:52.095481 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:52.095315 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 01:03:53.095485 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:53.095445 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:53.095994 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:53.095495 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 01:03:54.096029 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:54.095990 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:54.096524 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:54.096039 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 01:03:55.508654 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:55.508604 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:55.509255 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:55.509063 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:55.510001 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:03:55.509983 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:03:58.656052 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:03:58.656015 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:04:00.888095 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.888063 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2"] Nov 03 01:04:00.892032 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.892016 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:00.951500 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.951473 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2"] Nov 03 01:04:00.953605 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.953567 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:00.953743 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.953630 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:00.953743 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.953729 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52860790-c4c5-4643-896a-ea677cfc3495-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:00.953881 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.953758 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j6hn\" (UniqueName: \"kubernetes.io/projected/52860790-c4c5-4643-896a-ea677cfc3495-kube-api-access-7j6hn\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:00.953881 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.953857 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52860790-c4c5-4643-896a-ea677cfc3495-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:00.953991 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.953880 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52860790-c4c5-4643-896a-ea677cfc3495-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:00.953991 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.953938 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:00.953991 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.953971 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:00.954139 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:00.954071 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.055207 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.055179 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52860790-c4c5-4643-896a-ea677cfc3495-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.055394 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.055215 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j6hn\" (UniqueName: \"kubernetes.io/projected/52860790-c4c5-4643-896a-ea677cfc3495-kube-api-access-7j6hn\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.055394 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.055282 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52860790-c4c5-4643-896a-ea677cfc3495-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.055394 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.055304 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52860790-c4c5-4643-896a-ea677cfc3495-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.055394 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.055351 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.055607 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.055396 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.055607 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.055512 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.055607 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.055583 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.055758 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.055639 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.056110 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.056076 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52860790-c4c5-4643-896a-ea677cfc3495-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.056209 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.056118 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.056209 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.056080 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.056209 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.056180 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.056341 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.056229 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.057826 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.057768 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52860790-c4c5-4643-896a-ea677cfc3495-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.057974 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.057955 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52860790-c4c5-4643-896a-ea677cfc3495-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.111474 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.111444 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52860790-c4c5-4643-896a-ea677cfc3495-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.111579 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.111522 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j6hn\" (UniqueName: \"kubernetes.io/projected/52860790-c4c5-4643-896a-ea677cfc3495-kube-api-access-7j6hn\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m75z2\" (UID: \"52860790-c4c5-4643-896a-ea677cfc3495\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.206429 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.206401 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:01.631254 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.631233 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2"] Nov 03 01:04:01.633442 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:04:01.633417 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52860790_c4c5_4643_896a_ea677cfc3495.slice/crio-9c02430b24d7f3acf910e07ee658c81b425feb7a3b6c7d2b7a5ad3883ec1ec3c WatchSource:0}: Error finding container 9c02430b24d7f3acf910e07ee658c81b425feb7a3b6c7d2b7a5ad3883ec1ec3c: Status 404 returned error can't find the container with id 9c02430b24d7f3acf910e07ee658c81b425feb7a3b6c7d2b7a5ad3883ec1ec3c Nov 03 01:04:01.636002 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.635974 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:01.636070 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.636037 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:01.636070 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.636066 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:01.876346 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.876306 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" event={"ID":"52860790-c4c5-4643-896a-ea677cfc3495","Type":"ContainerStarted","Data":"e804088daba57f45c4f5f33d1c2294394906db02ae36fb37e9b7f960190abb67"} Nov 03 01:04:01.876514 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.876364 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" event={"ID":"52860790-c4c5-4643-896a-ea677cfc3495","Type":"ContainerStarted","Data":"9c02430b24d7f3acf910e07ee658c81b425feb7a3b6c7d2b7a5ad3883ec1ec3c"} Nov 03 01:04:01.995049 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:01.994994 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" podStartSLOduration=1.994974514 podStartE2EDuration="1.994974514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:01.991896681 +0000 UTC m=+3525.275454355" watchObservedRunningTime="2026-11-03 01:04:01.994974514 +0000 UTC m=+3525.278532186" Nov 03 01:04:02.207436 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:02.207399 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:02.209104 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:02.209084 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 01:04:02.209171 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:02.209131 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" podUID="52860790-c4c5-4643-896a-ea677cfc3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 01:04:03.207819 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:03.207756 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 01:04:03.208218 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:03.207831 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" podUID="52860790-c4c5-4643-896a-ea677cfc3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 01:04:04.207470 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:04.207436 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 01:04:04.207652 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:04.207490 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" podUID="52860790-c4c5-4643-896a-ea677cfc3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 01:04:05.206733 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:05.206693 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 01:04:05.207168 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:05.206754 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" podUID="52860790-c4c5-4643-896a-ea677cfc3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 01:04:06.207428 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:06.207392 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 01:04:06.207840 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:06.207453 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" podUID="52860790-c4c5-4643-896a-ea677cfc3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 01:04:07.207474 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:07.207441 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 01:04:07.207875 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:07.207501 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" podUID="52860790-c4c5-4643-896a-ea677cfc3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 01:04:08.206870 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:08.206835 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 01:04:08.207058 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:08.206917 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" podUID="52860790-c4c5-4643-896a-ea677cfc3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 01:04:10.207694 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:10.207661 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:04:10.208172 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:10.207727 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" podUID="52860790-c4c5-4643-896a-ea677cfc3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:10.232264 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:10.232232 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:10.232659 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:10.232637 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:10.233796 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:10.233750 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" Nov 03 01:04:20.088768 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.088738 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb"] Nov 03 01:04:20.093835 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.093811 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:04:20.149665 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.149635 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb"] Nov 03 01:04:20.201074 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.201039 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4bfaef\"" Nov 03 01:04:20.201384 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.201365 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4bfaef-dockercfg-6qh2b\"" Nov 03 01:04:20.242882 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.242834 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8"] Nov 03 01:04:20.248293 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.248260 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/032fe307-1e47-4c10-885e-10027dec0da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb\" (UID: \"032fe307-1e47-4c10-885e-10027dec0da7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:04:20.248831 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.248598 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gppsl\" (UniqueName: \"kubernetes.io/projected/032fe307-1e47-4c10-885e-10027dec0da7-kube-api-access-gppsl\") pod \"payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb\" (UID: \"032fe307-1e47-4c10-885e-10027dec0da7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:04:20.250222 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.249635 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:04:20.286179 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.286143 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8"] Nov 03 01:04:20.349573 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.349500 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8\" (UID: \"82f60f9c-168b-4f7e-b15a-4deee10e3eb2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:04:20.349705 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.349679 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkssg\" (UniqueName: \"kubernetes.io/projected/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-kube-api-access-wkssg\") pod \"payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8\" (UID: \"82f60f9c-168b-4f7e-b15a-4deee10e3eb2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:04:20.349748 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.349731 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/032fe307-1e47-4c10-885e-10027dec0da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb\" (UID: \"032fe307-1e47-4c10-885e-10027dec0da7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:04:20.349862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.349846 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gppsl\" (UniqueName: \"kubernetes.io/projected/032fe307-1e47-4c10-885e-10027dec0da7-kube-api-access-gppsl\") pod \"payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb\" (UID: \"032fe307-1e47-4c10-885e-10027dec0da7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:04:20.350353 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.350333 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/032fe307-1e47-4c10-885e-10027dec0da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb\" (UID: \"032fe307-1e47-4c10-885e-10027dec0da7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:04:20.406458 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.406434 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gppsl\" (UniqueName: \"kubernetes.io/projected/032fe307-1e47-4c10-885e-10027dec0da7-kube-api-access-gppsl\") pod \"payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb\" (UID: \"032fe307-1e47-4c10-885e-10027dec0da7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:04:20.408286 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.408265 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:04:20.451137 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.451094 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkssg\" (UniqueName: \"kubernetes.io/projected/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-kube-api-access-wkssg\") pod \"payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8\" (UID: \"82f60f9c-168b-4f7e-b15a-4deee10e3eb2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:04:20.451857 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.451829 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8\" (UID: \"82f60f9c-168b-4f7e-b15a-4deee10e3eb2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:04:20.452556 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.452533 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8\" (UID: \"82f60f9c-168b-4f7e-b15a-4deee10e3eb2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:04:20.508376 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.508343 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkssg\" (UniqueName: \"kubernetes.io/projected/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-kube-api-access-wkssg\") pod \"payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8\" (UID: \"82f60f9c-168b-4f7e-b15a-4deee10e3eb2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:04:20.567908 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.567878 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:04:20.784863 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.784835 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb"] Nov 03 01:04:20.785657 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:04:20.785622 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032fe307_1e47_4c10_885e_10027dec0da7.slice/crio-57f95f9928125af88f5c4cb4a951e17be30f17d284cec2e6c96588278f59b22e WatchSource:0}: Error finding container 57f95f9928125af88f5c4cb4a951e17be30f17d284cec2e6c96588278f59b22e: Status 404 returned error can't find the container with id 57f95f9928125af88f5c4cb4a951e17be30f17d284cec2e6c96588278f59b22e Nov 03 01:04:20.944948 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:20.944925 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8"] Nov 03 01:04:20.946350 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:04:20.946322 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f60f9c_168b_4f7e_b15a_4deee10e3eb2.slice/crio-1e9796b319eead0b735557f61d3ce5161f5732be19273a79bc69d3ec4901c747 WatchSource:0}: Error finding container 1e9796b319eead0b735557f61d3ce5161f5732be19273a79bc69d3ec4901c747: Status 404 returned error can't find the container with id 1e9796b319eead0b735557f61d3ce5161f5732be19273a79bc69d3ec4901c747 Nov 03 01:04:21.043464 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:21.043431 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" event={"ID":"82f60f9c-168b-4f7e-b15a-4deee10e3eb2","Type":"ContainerStarted","Data":"c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338"} Nov 03 01:04:21.043600 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:21.043499 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:04:21.043600 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:21.043524 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" event={"ID":"82f60f9c-168b-4f7e-b15a-4deee10e3eb2","Type":"ContainerStarted","Data":"1e9796b319eead0b735557f61d3ce5161f5732be19273a79bc69d3ec4901c747"} Nov 03 01:04:21.045106 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:21.045084 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" event={"ID":"032fe307-1e47-4c10-885e-10027dec0da7","Type":"ContainerStarted","Data":"2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e"} Nov 03 01:04:21.045213 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:21.045124 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" event={"ID":"032fe307-1e47-4c10-885e-10027dec0da7","Type":"ContainerStarted","Data":"57f95f9928125af88f5c4cb4a951e17be30f17d284cec2e6c96588278f59b22e"} Nov 03 01:04:21.045213 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:21.045153 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:04:21.157035 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:21.156994 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" podStartSLOduration=1.156981321 podStartE2EDuration="1.156981321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:21.154322626 +0000 UTC m=+3544.437880298" watchObservedRunningTime="2026-11-03 01:04:21.156981321 +0000 UTC m=+3544.440538999" Nov 03 01:04:21.269751 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:21.269620 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" podStartSLOduration=1.269606964 podStartE2EDuration="1.269606964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:21.268337796 +0000 UTC m=+3544.551895466" watchObservedRunningTime="2026-11-03 01:04:21.269606964 +0000 UTC m=+3544.553164636" Nov 03 01:04:28.655121 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:04:28.655079 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:04:32.055435 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:32.055360 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:04:32.056008 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:32.055728 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:04:37.736802 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.736746 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq"] Nov 03 01:04:37.740896 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.740872 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.800261 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.800228 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq"] Nov 03 01:04:37.844893 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.844865 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-67b917-openshift-default-dockercfg-z2tb2\"" Nov 03 01:04:37.846414 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.846395 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-envoy\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.846503 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.846423 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d949865-b1a9-4c64-909b-5d617c81496c-istiod-ca-cert\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.846567 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.846534 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-istio-token\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.846666 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.846646 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-socket\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.846716 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.846685 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d949865-b1a9-4c64-909b-5d617c81496c-istio-podinfo\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.846763 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.846729 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw5sl\" (UniqueName: \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-kube-api-access-fw5sl\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.846847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.846786 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-certs\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.846847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.846819 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-credential-socket\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.846847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.846840 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-data\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.993847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.992253 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-envoy\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.993847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.992304 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d949865-b1a9-4c64-909b-5d617c81496c-istiod-ca-cert\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.993847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.992388 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-istio-token\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.993847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.992453 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-socket\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.993847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.992489 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d949865-b1a9-4c64-909b-5d617c81496c-istio-podinfo\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.993847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.992519 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw5sl\" (UniqueName: \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-kube-api-access-fw5sl\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.993847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.992592 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-certs\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.993847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.992650 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-credential-socket\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.993847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.992677 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-data\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:37.993847 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:37.993209 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-data\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:38.023797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.005419 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d949865-b1a9-4c64-909b-5d617c81496c-istiod-ca-cert\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:38.049244 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.024323 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-certs\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:38.049547 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.025086 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-credential-socket\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:38.049653 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.025286 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-socket\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:38.071183 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.070538 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-envoy\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:38.072362 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.072268 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d949865-b1a9-4c64-909b-5d617c81496c-istio-podinfo\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:38.089430 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.089396 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-istio-token\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:38.091149 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.091128 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw5sl\" (UniqueName: \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-kube-api-access-fw5sl\") pod \"e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:38.355225 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.355138 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:38.782841 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.782816 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq"] Nov 03 01:04:38.784827 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:04:38.784801 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d949865_b1a9_4c64_909b_5d617c81496c.slice/crio-2497899b00de85d63bb4cd343cacde3f0004584fd4d608d813980264beced42c WatchSource:0}: Error finding container 2497899b00de85d63bb4cd343cacde3f0004584fd4d608d813980264beced42c: Status 404 returned error can't find the container with id 2497899b00de85d63bb4cd343cacde3f0004584fd4d608d813980264beced42c Nov 03 01:04:38.787380 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.787350 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:38.787453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.787435 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:38.787489 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:38.787464 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:39.162866 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:39.162830 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" event={"ID":"6d949865-b1a9-4c64-909b-5d617c81496c","Type":"ContainerStarted","Data":"181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62"} Nov 03 01:04:39.163035 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:39.162925 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" event={"ID":"6d949865-b1a9-4c64-909b-5d617c81496c","Type":"ContainerStarted","Data":"2497899b00de85d63bb4cd343cacde3f0004584fd4d608d813980264beced42c"} Nov 03 01:04:39.331027 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:39.330976 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" podStartSLOduration=2.330959255 podStartE2EDuration="2.330959255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:39.328475581 +0000 UTC m=+3562.612033254" watchObservedRunningTime="2026-11-03 01:04:39.330959255 +0000 UTC m=+3562.614516927" Nov 03 01:04:39.355926 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:39.355893 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:39.361759 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:39.361737 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:40.171021 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:40.170990 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:40.171884 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:40.171867 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:04:56.376269 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.376232 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c"] Nov 03 01:04:56.380297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.380279 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:56.434490 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.434464 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4db6f5-serving-cert\"" Nov 03 01:04:56.487350 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.487321 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c"] Nov 03 01:04:56.532618 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.532579 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-metrics-certs\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:56.532790 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.532644 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-maas-api-tls\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:56.532843 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.532824 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd5b850-54fd-4bed-aefa-c0b60f902c44-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:56.532885 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.532873 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvm5n\" (UniqueName: \"kubernetes.io/projected/6bd5b850-54fd-4bed-aefa-c0b60f902c44-kube-api-access-zvm5n\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:56.634114 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.634028 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd5b850-54fd-4bed-aefa-c0b60f902c44-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:56.634114 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.634080 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvm5n\" (UniqueName: \"kubernetes.io/projected/6bd5b850-54fd-4bed-aefa-c0b60f902c44-kube-api-access-zvm5n\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:56.634362 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.634131 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-metrics-certs\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:56.634362 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.634169 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-maas-api-tls\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:56.634362 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:04:56.634321 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-serving-cert: secret "maas-api-e2e-inf-a-4db6f5-serving-cert" not found Nov 03 01:04:56.634540 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:04:56.634406 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-maas-api-tls podName:6bd5b850-54fd-4bed-aefa-c0b60f902c44 nodeName:}" failed. No retries permitted until 2026-11-03 01:04:57.134385637 +0000 UTC m=+3580.417943302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-maas-api-tls") pod "maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" (UID: "6bd5b850-54fd-4bed-aefa-c0b60f902c44") : secret "maas-api-e2e-inf-a-4db6f5-serving-cert" not found Nov 03 01:04:56.634828 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.634806 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd5b850-54fd-4bed-aefa-c0b60f902c44-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:56.636677 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.636659 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-metrics-certs\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:56.691411 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:56.691377 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvm5n\" (UniqueName: \"kubernetes.io/projected/6bd5b850-54fd-4bed-aefa-c0b60f902c44-kube-api-access-zvm5n\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:57.140145 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:57.140111 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-maas-api-tls\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:57.142593 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:57.142575 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-maas-api-tls\") pod \"maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:57.293839 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:57.293816 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:57.677094 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:57.677070 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c"] Nov 03 01:04:57.678459 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:04:57.678436 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd5b850_54fd_4bed_aefa_c0b60f902c44.slice/crio-708acdf2d7c570b5a7ad2511df648e9978b2d09fd819d8efccca6488a9369c77 WatchSource:0}: Error finding container 708acdf2d7c570b5a7ad2511df648e9978b2d09fd819d8efccca6488a9369c77: Status 404 returned error can't find the container with id 708acdf2d7c570b5a7ad2511df648e9978b2d09fd819d8efccca6488a9369c77 Nov 03 01:04:57.679906 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:57.679886 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:04:58.297994 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:58.297959 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" event={"ID":"6bd5b850-54fd-4bed-aefa-c0b60f902c44","Type":"ContainerStarted","Data":"708acdf2d7c570b5a7ad2511df648e9978b2d09fd819d8efccca6488a9369c77"} Nov 03 01:04:58.654687 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:04:58.654650 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:04:59.306161 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:59.306118 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" event={"ID":"6bd5b850-54fd-4bed-aefa-c0b60f902c44","Type":"ContainerStarted","Data":"97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421"} Nov 03 01:04:59.306552 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:59.306198 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:04:59.429861 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:04:59.423476 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" podStartSLOduration=2.559208203 podStartE2EDuration="3.423455694s" totalImagesPullingTime="864.247491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:59.419210632 +0000 UTC m=+3582.702768331" watchObservedRunningTime="2026-11-03 01:04:59.423455694 +0000 UTC m=+3582.707013367" Nov 03 01:05:05.317543 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:05.317515 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:05:28.656943 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:05:28.656899 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:05:28.785034 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:28.784988 4304 patch_prober.go:29] interesting pod/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:05:28.785238 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:28.785091 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:31.581884 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:31.581850 4304 generic.go:356] "Generic (PLEG): container finished" podID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerID="866f977aa251107cea02af898260fc7dd824cc29c85825b97d984c955832fca1" exitCode=0 Nov 03 01:05:31.582301 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:31.581928 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" event={"ID":"e70c8d9a-48b8-48f2-b985-001e2448f741","Type":"ContainerDied","Data":"866f977aa251107cea02af898260fc7dd824cc29c85825b97d984c955832fca1"} Nov 03 01:05:31.582351 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:31.582305 4304 scope.go:122] "RemoveContainer" containerID="866f977aa251107cea02af898260fc7dd824cc29c85825b97d984c955832fca1" Nov 03 01:05:31.583899 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:31.583872 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:05:31.583958 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:31.583949 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:05:31.584001 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:31.583985 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:05:32.593523 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:32.593490 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" event={"ID":"e70c8d9a-48b8-48f2-b985-001e2448f741","Type":"ContainerStarted","Data":"98e39dfbcbc9bb032f4dce30db690a191eba9e43c5ad40eff86f437086c664a8"} Nov 03 01:05:32.818083 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:32.818053 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:05:32.818244 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:32.818210 4304 patch_prober.go:29] interesting pod/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 01:05:32.818287 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:32.818264 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 01:05:33.817961 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:33.817929 4304 patch_prober.go:29] interesting pod/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 01:05:33.818374 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:33.817979 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 01:05:34.818251 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:34.818209 4304 patch_prober.go:29] interesting pod/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 01:05:34.818736 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:34.818270 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 01:05:35.818313 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:35.818277 4304 patch_prober.go:29] interesting pod/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 01:05:35.818718 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:35.818356 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 01:05:36.818887 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:36.818856 4304 patch_prober.go:29] interesting pod/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 01:05:36.819377 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:36.818912 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 01:05:37.818093 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:37.818060 4304 patch_prober.go:29] interesting pod/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 01:05:37.818293 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:37.818114 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 01:05:38.829792 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:38.829749 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:05:38.830243 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:38.830225 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:05:38.830885 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:05:38.830867 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:05:58.655126 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:05:58.655090 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:06:12.324381 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.324344 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q"] Nov 03 01:06:12.327264 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.327246 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.381931 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.381904 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-244924-serving-cert\"" Nov 03 01:06:12.389533 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.389506 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e863b69-239a-4f2f-974e-18dddebcdd1b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.389666 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.389569 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhg5j\" (UniqueName: \"kubernetes.io/projected/6e863b69-239a-4f2f-974e-18dddebcdd1b-kube-api-access-bhg5j\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.389666 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.389645 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-maas-api-tls\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.389815 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.389726 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-metrics-certs\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.434709 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.434687 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q"] Nov 03 01:06:12.491402 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.491364 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-maas-api-tls\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.491402 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.491400 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-metrics-certs\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.491580 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:06:12.491519 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-serving-cert: secret "maas-api-e2e-inf-a-244924-serving-cert" not found Nov 03 01:06:12.491580 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.491568 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e863b69-239a-4f2f-974e-18dddebcdd1b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.491580 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:06:12.491579 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-maas-api-tls podName:6e863b69-239a-4f2f-974e-18dddebcdd1b nodeName:}" failed. No retries permitted until 2026-11-03 01:06:12.991563515 +0000 UTC m=+3656.275121170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-maas-api-tls") pod "maas-api-e2e-inf-a-244924-766c67dc49-cph6q" (UID: "6e863b69-239a-4f2f-974e-18dddebcdd1b") : secret "maas-api-e2e-inf-a-244924-serving-cert" not found Nov 03 01:06:12.491703 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.491645 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhg5j\" (UniqueName: \"kubernetes.io/projected/6e863b69-239a-4f2f-974e-18dddebcdd1b-kube-api-access-bhg5j\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.492311 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.492243 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e863b69-239a-4f2f-974e-18dddebcdd1b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.494041 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.494019 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-metrics-certs\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.549956 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.549924 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhg5j\" (UniqueName: \"kubernetes.io/projected/6e863b69-239a-4f2f-974e-18dddebcdd1b-kube-api-access-bhg5j\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:12.999588 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:12.999553 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-maas-api-tls\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:13.002080 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:13.002060 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-maas-api-tls\") pod \"maas-api-e2e-inf-a-244924-766c67dc49-cph6q\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:13.239735 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:13.239705 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:13.619976 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:13.619954 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q"] Nov 03 01:06:13.621807 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:06:13.621758 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e863b69_239a_4f2f_974e_18dddebcdd1b.slice/crio-2ee438b1a451d74a6eae153853290f0e4ef7e7a029df52805ab022922fa86c05 WatchSource:0}: Error finding container 2ee438b1a451d74a6eae153853290f0e4ef7e7a029df52805ab022922fa86c05: Status 404 returned error can't find the container with id 2ee438b1a451d74a6eae153853290f0e4ef7e7a029df52805ab022922fa86c05 Nov 03 01:06:13.863274 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:13.863240 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" event={"ID":"6e863b69-239a-4f2f-974e-18dddebcdd1b","Type":"ContainerStarted","Data":"2ee438b1a451d74a6eae153853290f0e4ef7e7a029df52805ab022922fa86c05"} Nov 03 01:06:14.871519 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:14.871482 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" event={"ID":"6e863b69-239a-4f2f-974e-18dddebcdd1b","Type":"ContainerStarted","Data":"25809f4e0904614e308857464824bf3d6531cb7029478fd949794fc945b5c779"} Nov 03 01:06:14.871904 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:14.871567 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:14.988202 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:14.988149 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" podStartSLOduration=2.227177576 podStartE2EDuration="2.988137428s" totalImagesPullingTime="760.959852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:14.985156522 +0000 UTC m=+3658.268714194" watchObservedRunningTime="2026-11-03 01:06:14.988137428 +0000 UTC m=+3658.271695101" Nov 03 01:06:20.883838 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:20.883805 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:06:26.211919 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:26.211884 4304 patch_prober.go:29] interesting pod/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:06:26.212328 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:26.211961 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:28.655130 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:06:28.655094 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:06:28.986630 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:28.986556 4304 generic.go:356] "Generic (PLEG): container finished" podID="6d949865-b1a9-4c64-909b-5d617c81496c" containerID="181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62" exitCode=0 Nov 03 01:06:28.986630 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:28.986616 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" event={"ID":"6d949865-b1a9-4c64-909b-5d617c81496c","Type":"ContainerDied","Data":"181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62"} Nov 03 01:06:28.987041 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:28.987023 4304 scope.go:122] "RemoveContainer" containerID="181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62" Nov 03 01:06:28.988588 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:28.988556 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:06:28.988707 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:28.988640 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:06:28.988707 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:28.988687 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:06:29.995181 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:29.995141 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" event={"ID":"6d949865-b1a9-4c64-909b-5d617c81496c","Type":"ContainerStarted","Data":"35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996"} Nov 03 01:06:30.356459 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:30.356270 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:06:30.357080 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:30.357048 4304 patch_prober.go:29] interesting pod/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 01:06:30.357229 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:30.357108 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 01:06:31.356384 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:31.356349 4304 patch_prober.go:29] interesting pod/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 01:06:31.356840 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:31.356411 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 01:06:32.355988 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:32.355952 4304 patch_prober.go:29] interesting pod/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 01:06:32.356218 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:32.356020 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 01:06:33.356287 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:33.356249 4304 patch_prober.go:29] interesting pod/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 01:06:33.356681 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:33.356301 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 01:06:34.355882 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:34.355839 4304 patch_prober.go:29] interesting pod/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 01:06:34.356061 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:34.355904 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 01:06:35.360371 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:35.360340 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:06:35.360906 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:35.360883 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:06:35.361540 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:06:35.361520 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:06:58.655356 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:06:58.655313 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:07:28.654524 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:07:28.654489 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:07:58.656174 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:07:58.656093 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:08:28.655360 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:08:28.655326 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:08:44.285044 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:44.285014 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8"] Nov 03 01:08:44.398410 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:44.398374 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb"] Nov 03 01:08:49.286034 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.285980 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" podUID="82f60f9c-168b-4f7e-b15a-4deee10e3eb2" containerName="payload-processing" containerID="cri-o://c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338" gracePeriod=25 Nov 03 01:08:49.399423 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.399379 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" podUID="032fe307-1e47-4c10-885e-10027dec0da7" containerName="payload-pre-processing" containerID="cri-o://2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e" gracePeriod=25 Nov 03 01:08:49.555918 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.555858 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:08:49.604343 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.604314 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-kube-api-access-wkssg\" (UniqueName: \"kubernetes.io/projected/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-kube-api-access-wkssg\") pod \"82f60f9c-168b-4f7e-b15a-4deee10e3eb2\" (UID: \"82f60f9c-168b-4f7e-b15a-4deee10e3eb2\") " Nov 03 01:08:49.604495 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.604484 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-plugins-config-volume\") pod \"82f60f9c-168b-4f7e-b15a-4deee10e3eb2\" (UID: \"82f60f9c-168b-4f7e-b15a-4deee10e3eb2\") " Nov 03 01:08:49.605016 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.604990 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-plugins-config-volume" pod "82f60f9c-168b-4f7e-b15a-4deee10e3eb2" (UID: "82f60f9c-168b-4f7e-b15a-4deee10e3eb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:08:49.607140 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.607097 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-kube-api-access-wkssg" pod "82f60f9c-168b-4f7e-b15a-4deee10e3eb2" (UID: "82f60f9c-168b-4f7e-b15a-4deee10e3eb2"). InnerVolumeSpecName "kube-api-access-wkssg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:08:49.659362 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.659341 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:08:49.705919 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.705882 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032fe307-1e47-4c10-885e-10027dec0da7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/032fe307-1e47-4c10-885e-10027dec0da7-plugins-config-volume\") pod \"032fe307-1e47-4c10-885e-10027dec0da7\" (UID: \"032fe307-1e47-4c10-885e-10027dec0da7\") " Nov 03 01:08:49.706085 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.706056 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032fe307-1e47-4c10-885e-10027dec0da7-kube-api-access-gppsl\" (UniqueName: \"kubernetes.io/projected/032fe307-1e47-4c10-885e-10027dec0da7-kube-api-access-gppsl\") pod \"032fe307-1e47-4c10-885e-10027dec0da7\" (UID: \"032fe307-1e47-4c10-885e-10027dec0da7\") " Nov 03 01:08:49.706247 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.706224 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032fe307-1e47-4c10-885e-10027dec0da7-plugins-config-volume" pod "032fe307-1e47-4c10-885e-10027dec0da7" (UID: "032fe307-1e47-4c10-885e-10027dec0da7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:08:49.706529 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.706510 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkssg\" (UniqueName: \"kubernetes.io/projected/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-kube-api-access-wkssg\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:49.706596 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.706534 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/032fe307-1e47-4c10-885e-10027dec0da7-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:49.706596 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.706551 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f60f9c-168b-4f7e-b15a-4deee10e3eb2-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:49.708213 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.708190 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032fe307-1e47-4c10-885e-10027dec0da7-kube-api-access-gppsl" pod "032fe307-1e47-4c10-885e-10027dec0da7" (UID: "032fe307-1e47-4c10-885e-10027dec0da7"). InnerVolumeSpecName "kube-api-access-gppsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:08:49.808160 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.808096 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gppsl\" (UniqueName: \"kubernetes.io/projected/032fe307-1e47-4c10-885e-10027dec0da7-kube-api-access-gppsl\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:49.912227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.912192 4304 generic.go:356] "Generic (PLEG): container finished" podID="82f60f9c-168b-4f7e-b15a-4deee10e3eb2" containerID="c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338" exitCode=0 Nov 03 01:08:49.912372 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.912233 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" Nov 03 01:08:49.912372 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.912266 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" event={"ID":"82f60f9c-168b-4f7e-b15a-4deee10e3eb2","Type":"ContainerDied","Data":"c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338"} Nov 03 01:08:49.912372 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.912327 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8" event={"ID":"82f60f9c-168b-4f7e-b15a-4deee10e3eb2","Type":"ContainerDied","Data":"1e9796b319eead0b735557f61d3ce5161f5732be19273a79bc69d3ec4901c747"} Nov 03 01:08:49.912372 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.912355 4304 scope.go:122] "RemoveContainer" containerID="c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338" Nov 03 01:08:49.914080 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.914060 4304 generic.go:356] "Generic (PLEG): container finished" podID="032fe307-1e47-4c10-885e-10027dec0da7" containerID="2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e" exitCode=0 Nov 03 01:08:49.914213 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.914126 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" Nov 03 01:08:49.914213 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.914134 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" event={"ID":"032fe307-1e47-4c10-885e-10027dec0da7","Type":"ContainerDied","Data":"2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e"} Nov 03 01:08:49.914213 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.914195 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb" event={"ID":"032fe307-1e47-4c10-885e-10027dec0da7","Type":"ContainerDied","Data":"57f95f9928125af88f5c4cb4a951e17be30f17d284cec2e6c96588278f59b22e"} Nov 03 01:08:49.923345 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.923330 4304 scope.go:122] "RemoveContainer" containerID="c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338" Nov 03 01:08:49.923591 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:08:49.923573 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338\": container with ID starting with c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338 not found: ID does not exist" containerID="c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338" Nov 03 01:08:49.923681 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.923598 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338"} err="failed to get container status \"c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338\": rpc error: code = NotFound desc = could not find container \"c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338\": container with ID starting with c56c23e1ec2babf51a0648713ed1bc9ec44f96c0db630c73b4271a3d2e1bd338 not found: ID does not exist" Nov 03 01:08:49.923681 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.923618 4304 scope.go:122] "RemoveContainer" containerID="2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e" Nov 03 01:08:49.932929 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.932913 4304 scope.go:122] "RemoveContainer" containerID="2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e" Nov 03 01:08:49.933178 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:08:49.933161 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e\": container with ID starting with 2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e not found: ID does not exist" containerID="2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e" Nov 03 01:08:49.933225 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:49.933187 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e"} err="failed to get container status \"2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e\": rpc error: code = NotFound desc = could not find container \"2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e\": container with ID starting with 2e23bf179f59573631b6039d0e7b3a6d7089c4a2b3b739250d32fe4619d7673e not found: ID does not exist" Nov 03 01:08:50.075346 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:50.075282 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8"] Nov 03 01:08:50.079930 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:50.079909 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4bfaef-58d74485b5-nd9k8"] Nov 03 01:08:50.186594 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:50.186571 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb"] Nov 03 01:08:50.189525 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:50.189505 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4bfaef-5959688c8c-nsgfb"] Nov 03 01:08:50.830640 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:50.830605 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032fe307-1e47-4c10-885e-10027dec0da7" path="/var/lib/kubelet/pods/032fe307-1e47-4c10-885e-10027dec0da7/volumes" Nov 03 01:08:50.831039 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:08:50.831025 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f60f9c-168b-4f7e-b15a-4deee10e3eb2" path="/var/lib/kubelet/pods/82f60f9c-168b-4f7e-b15a-4deee10e3eb2/volumes" Nov 03 01:08:58.655913 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:08:58.655877 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:09:12.809395 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:12.809364 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q"] Nov 03 01:09:12.809784 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:12.809569 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" podUID="6e863b69-239a-4f2f-974e-18dddebcdd1b" containerName="maas-api" containerID="cri-o://25809f4e0904614e308857464824bf3d6531cb7029478fd949794fc945b5c779" gracePeriod=30 Nov 03 01:09:13.071947 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.071872 4304 generic.go:356] "Generic (PLEG): container finished" podID="6e863b69-239a-4f2f-974e-18dddebcdd1b" containerID="25809f4e0904614e308857464824bf3d6531cb7029478fd949794fc945b5c779" exitCode=0 Nov 03 01:09:13.072248 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.071956 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" event={"ID":"6e863b69-239a-4f2f-974e-18dddebcdd1b","Type":"ContainerDied","Data":"25809f4e0904614e308857464824bf3d6531cb7029478fd949794fc945b5c779"} Nov 03 01:09:13.104422 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.104396 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:09:13.256249 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.256207 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e863b69-239a-4f2f-974e-18dddebcdd1b-kube-api-access-bhg5j\" (UniqueName: \"kubernetes.io/projected/6e863b69-239a-4f2f-974e-18dddebcdd1b-kube-api-access-bhg5j\") pod \"6e863b69-239a-4f2f-974e-18dddebcdd1b\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " Nov 03 01:09:13.256447 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.256259 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-metrics-certs\") pod \"6e863b69-239a-4f2f-974e-18dddebcdd1b\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " Nov 03 01:09:13.256447 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.256402 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e863b69-239a-4f2f-974e-18dddebcdd1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e863b69-239a-4f2f-974e-18dddebcdd1b-openshift-service-ca\") pod \"6e863b69-239a-4f2f-974e-18dddebcdd1b\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " Nov 03 01:09:13.256589 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.256540 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-maas-api-tls\") pod \"6e863b69-239a-4f2f-974e-18dddebcdd1b\" (UID: \"6e863b69-239a-4f2f-974e-18dddebcdd1b\") " Nov 03 01:09:13.256905 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.256872 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e863b69-239a-4f2f-974e-18dddebcdd1b-openshift-service-ca" pod "6e863b69-239a-4f2f-974e-18dddebcdd1b" (UID: "6e863b69-239a-4f2f-974e-18dddebcdd1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:13.257525 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.257495 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e863b69-239a-4f2f-974e-18dddebcdd1b-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:13.258910 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.258881 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-maas-api-tls" pod "6e863b69-239a-4f2f-974e-18dddebcdd1b" (UID: "6e863b69-239a-4f2f-974e-18dddebcdd1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:09:13.259072 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.259055 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e863b69-239a-4f2f-974e-18dddebcdd1b-kube-api-access-bhg5j" pod "6e863b69-239a-4f2f-974e-18dddebcdd1b" (UID: "6e863b69-239a-4f2f-974e-18dddebcdd1b"). InnerVolumeSpecName "kube-api-access-bhg5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:13.259155 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.259127 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-metrics-certs" pod "6e863b69-239a-4f2f-974e-18dddebcdd1b" (UID: "6e863b69-239a-4f2f-974e-18dddebcdd1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:09:13.314826 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.313205 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq"] Nov 03 01:09:13.314826 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.313433 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" containerID="cri-o://35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996" gracePeriod=30 Nov 03 01:09:13.358960 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.358895 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhg5j\" (UniqueName: \"kubernetes.io/projected/6e863b69-239a-4f2f-974e-18dddebcdd1b-kube-api-access-bhg5j\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:13.358960 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.358921 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:13.358960 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:13.358932 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e863b69-239a-4f2f-974e-18dddebcdd1b-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:14.082123 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:14.082079 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" event={"ID":"6e863b69-239a-4f2f-974e-18dddebcdd1b","Type":"ContainerDied","Data":"2ee438b1a451d74a6eae153853290f0e4ef7e7a029df52805ab022922fa86c05"} Nov 03 01:09:14.082123 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:14.082117 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q" Nov 03 01:09:14.082585 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:14.082152 4304 scope.go:122] "RemoveContainer" containerID="25809f4e0904614e308857464824bf3d6531cb7029478fd949794fc945b5c779" Nov 03 01:09:14.246012 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:14.245981 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q"] Nov 03 01:09:14.295563 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:14.295537 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-244924-766c67dc49-cph6q"] Nov 03 01:09:14.829932 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:14.829900 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e863b69-239a-4f2f-974e-18dddebcdd1b" path="/var/lib/kubelet/pods/6e863b69-239a-4f2f-974e-18dddebcdd1b/volumes" Nov 03 01:09:18.572248 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.572225 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:09:18.644684 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.644656 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-credential-socket\") pod \"6d949865-b1a9-4c64-909b-5d617c81496c\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " Nov 03 01:09:18.644851 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.644720 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-certs\") pod \"6d949865-b1a9-4c64-909b-5d617c81496c\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " Nov 03 01:09:18.644851 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.644755 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-socket\") pod \"6d949865-b1a9-4c64-909b-5d617c81496c\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " Nov 03 01:09:18.644851 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.644797 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-kube-api-access-fw5sl\" (UniqueName: \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-kube-api-access-fw5sl\") pod \"6d949865-b1a9-4c64-909b-5d617c81496c\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " Nov 03 01:09:18.644851 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.644837 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d949865-b1a9-4c64-909b-5d617c81496c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d949865-b1a9-4c64-909b-5d617c81496c-istiod-ca-cert\") pod \"6d949865-b1a9-4c64-909b-5d617c81496c\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " Nov 03 01:09:18.645089 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.644884 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-envoy\") pod \"6d949865-b1a9-4c64-909b-5d617c81496c\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " Nov 03 01:09:18.645089 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.644940 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-data\") pod \"6d949865-b1a9-4c64-909b-5d617c81496c\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " Nov 03 01:09:18.645089 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.644959 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-credential-socket" pod "6d949865-b1a9-4c64-909b-5d617c81496c" (UID: "6d949865-b1a9-4c64-909b-5d617c81496c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:18.645089 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.645002 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-istio-token\" (UniqueName: \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-istio-token\") pod \"6d949865-b1a9-4c64-909b-5d617c81496c\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " Nov 03 01:09:18.645089 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.645041 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-certs" pod "6d949865-b1a9-4c64-909b-5d617c81496c" (UID: "6d949865-b1a9-4c64-909b-5d617c81496c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:18.645089 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.645067 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-socket" pod "6d949865-b1a9-4c64-909b-5d617c81496c" (UID: "6d949865-b1a9-4c64-909b-5d617c81496c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:18.645408 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.645228 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d949865-b1a9-4c64-909b-5d617c81496c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d949865-b1a9-4c64-909b-5d617c81496c-istio-podinfo\") pod \"6d949865-b1a9-4c64-909b-5d617c81496c\" (UID: \"6d949865-b1a9-4c64-909b-5d617c81496c\") " Nov 03 01:09:18.645408 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.645270 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d949865-b1a9-4c64-909b-5d617c81496c-istiod-ca-cert" pod "6d949865-b1a9-4c64-909b-5d617c81496c" (UID: "6d949865-b1a9-4c64-909b-5d617c81496c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:18.645408 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.645282 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-data" pod "6d949865-b1a9-4c64-909b-5d617c81496c" (UID: "6d949865-b1a9-4c64-909b-5d617c81496c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:18.645940 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.645918 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:18.646046 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.645945 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:18.646046 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.645963 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:18.646046 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.645978 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:18.646046 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.645993 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d949865-b1a9-4c64-909b-5d617c81496c-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:18.647326 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.647305 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-envoy" pod "6d949865-b1a9-4c64-909b-5d617c81496c" (UID: "6d949865-b1a9-4c64-909b-5d617c81496c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:18.647415 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.647381 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-kube-api-access-fw5sl" pod "6d949865-b1a9-4c64-909b-5d617c81496c" (UID: "6d949865-b1a9-4c64-909b-5d617c81496c"). InnerVolumeSpecName "kube-api-access-fw5sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:18.647668 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.647653 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-istio-token" pod "6d949865-b1a9-4c64-909b-5d617c81496c" (UID: "6d949865-b1a9-4c64-909b-5d617c81496c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:18.647727 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.647684 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d949865-b1a9-4c64-909b-5d617c81496c-istio-podinfo" pod "6d949865-b1a9-4c64-909b-5d617c81496c" (UID: "6d949865-b1a9-4c64-909b-5d617c81496c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:09:18.747000 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.746974 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw5sl\" (UniqueName: \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-kube-api-access-fw5sl\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:18.747112 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.746998 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d949865-b1a9-4c64-909b-5d617c81496c-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:18.747112 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.747021 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d949865-b1a9-4c64-909b-5d617c81496c-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:18.747112 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:18.747031 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d949865-b1a9-4c64-909b-5d617c81496c-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:19.117437 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.117345 4304 generic.go:356] "Generic (PLEG): container finished" podID="6d949865-b1a9-4c64-909b-5d617c81496c" containerID="35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996" exitCode=0 Nov 03 01:09:19.117437 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.117402 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" Nov 03 01:09:19.117437 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.117425 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" event={"ID":"6d949865-b1a9-4c64-909b-5d617c81496c","Type":"ContainerDied","Data":"35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996"} Nov 03 01:09:19.117714 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.117478 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq" event={"ID":"6d949865-b1a9-4c64-909b-5d617c81496c","Type":"ContainerDied","Data":"2497899b00de85d63bb4cd343cacde3f0004584fd4d608d813980264beced42c"} Nov 03 01:09:19.117714 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.117500 4304 scope.go:122] "RemoveContainer" containerID="35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996" Nov 03 01:09:19.128690 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.128674 4304 scope.go:122] "RemoveContainer" containerID="181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62" Nov 03 01:09:19.137498 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.137475 4304 scope.go:122] "RemoveContainer" containerID="35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996" Nov 03 01:09:19.137738 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:09:19.137720 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996\": container with ID starting with 35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996 not found: ID does not exist" containerID="35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996" Nov 03 01:09:19.137903 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.137746 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996"} err="failed to get container status \"35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996\": rpc error: code = NotFound desc = could not find container \"35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996\": container with ID starting with 35adafcee44fe1b55a81ad6487d31e45426c8f89995d7ec7d96fdb253bd5d996 not found: ID does not exist" Nov 03 01:09:19.137903 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.137763 4304 scope.go:122] "RemoveContainer" containerID="181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62" Nov 03 01:09:19.138015 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:09:19.138001 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62\": container with ID starting with 181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62 not found: ID does not exist" containerID="181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62" Nov 03 01:09:19.138054 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.138018 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62"} err="failed to get container status \"181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62\": rpc error: code = NotFound desc = could not find container \"181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62\": container with ID starting with 181172399cf9ac2c3bca842dd55fbb794c0b9b0ec299eb78770b3b43314f5b62 not found: ID does not exist" Nov 03 01:09:19.281927 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.281894 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq"] Nov 03 01:09:19.331693 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:19.331665 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-67b917-openshift-default-bf9b47bc9-rg9wq"] Nov 03 01:09:20.832471 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:09:20.832435 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" path="/var/lib/kubelet/pods/6d949865-b1a9-4c64-909b-5d617c81496c/volumes" Nov 03 01:09:28.655086 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:09:28.655044 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:09:58.656059 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:09:58.655996 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:10:28.654084 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:10:28.654043 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:10:42.950728 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:42.950520 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c"] Nov 03 01:10:42.951549 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:42.951514 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" podUID="6bd5b850-54fd-4bed-aefa-c0b60f902c44" containerName="maas-api" containerID="cri-o://97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421" gracePeriod=30 Nov 03 01:10:43.355156 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.355128 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:10:43.506961 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.503282 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd5b850-54fd-4bed-aefa-c0b60f902c44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd5b850-54fd-4bed-aefa-c0b60f902c44-openshift-service-ca\") pod \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " Nov 03 01:10:43.506961 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.503365 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-maas-api-tls\") pod \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " Nov 03 01:10:43.506961 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.503404 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-metrics-certs\") pod \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " Nov 03 01:10:43.506961 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.503543 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd5b850-54fd-4bed-aefa-c0b60f902c44-kube-api-access-zvm5n\" (UniqueName: \"kubernetes.io/projected/6bd5b850-54fd-4bed-aefa-c0b60f902c44-kube-api-access-zvm5n\") pod \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\" (UID: \"6bd5b850-54fd-4bed-aefa-c0b60f902c44\") " Nov 03 01:10:43.506961 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.504843 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd5b850-54fd-4bed-aefa-c0b60f902c44-openshift-service-ca" pod "6bd5b850-54fd-4bed-aefa-c0b60f902c44" (UID: "6bd5b850-54fd-4bed-aefa-c0b60f902c44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:43.596931 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.558079 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-metrics-certs" pod "6bd5b850-54fd-4bed-aefa-c0b60f902c44" (UID: "6bd5b850-54fd-4bed-aefa-c0b60f902c44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:43.596931 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.574247 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-maas-api-tls" pod "6bd5b850-54fd-4bed-aefa-c0b60f902c44" (UID: "6bd5b850-54fd-4bed-aefa-c0b60f902c44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:43.603288 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.603218 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd5b850-54fd-4bed-aefa-c0b60f902c44-kube-api-access-zvm5n" pod "6bd5b850-54fd-4bed-aefa-c0b60f902c44" (UID: "6bd5b850-54fd-4bed-aefa-c0b60f902c44"). InnerVolumeSpecName "kube-api-access-zvm5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:43.607016 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.604982 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvm5n\" (UniqueName: \"kubernetes.io/projected/6bd5b850-54fd-4bed-aefa-c0b60f902c44-kube-api-access-zvm5n\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:43.607016 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.605011 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd5b850-54fd-4bed-aefa-c0b60f902c44-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:43.607016 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.605027 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:43.607016 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.605038 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd5b850-54fd-4bed-aefa-c0b60f902c44-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:43.675797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.674932 4304 generic.go:356] "Generic (PLEG): container finished" podID="6bd5b850-54fd-4bed-aefa-c0b60f902c44" containerID="97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421" exitCode=0 Nov 03 01:10:43.675797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.674992 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" event={"ID":"6bd5b850-54fd-4bed-aefa-c0b60f902c44","Type":"ContainerDied","Data":"97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421"} Nov 03 01:10:43.675797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.675060 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" event={"ID":"6bd5b850-54fd-4bed-aefa-c0b60f902c44","Type":"ContainerDied","Data":"708acdf2d7c570b5a7ad2511df648e9978b2d09fd819d8efccca6488a9369c77"} Nov 03 01:10:43.675797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.675093 4304 scope.go:122] "RemoveContainer" containerID="97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421" Nov 03 01:10:43.675797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.675254 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c" Nov 03 01:10:43.695731 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.695664 4304 scope.go:122] "RemoveContainer" containerID="97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421" Nov 03 01:10:43.695956 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:10:43.695936 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421\": container with ID starting with 97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421 not found: ID does not exist" containerID="97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421" Nov 03 01:10:43.696016 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.695965 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421"} err="failed to get container status \"97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421\": rpc error: code = NotFound desc = could not find container \"97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421\": container with ID starting with 97fccc98bd93b87ba610aa5096f9c3294f6c230605be03a9ed1a4b3d15955421 not found: ID does not exist" Nov 03 01:10:43.869855 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.869545 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c"] Nov 03 01:10:43.908128 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:43.908099 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4db6f5-5465f4bf79-q5j6c"] Nov 03 01:10:44.886801 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:44.883597 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd5b850-54fd-4bed-aefa-c0b60f902c44" path="/var/lib/kubelet/pods/6bd5b850-54fd-4bed-aefa-c0b60f902c44/volumes" Nov 03 01:10:53.663983 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:53.663954 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8"] Nov 03 01:10:53.664407 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:53.664149 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" containerID="cri-o://98e39dfbcbc9bb032f4dce30db690a191eba9e43c5ad40eff86f437086c664a8" gracePeriod=30 Nov 03 01:10:53.833040 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:53.833007 4304 patch_prober.go:29] interesting pod/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:10:53.833241 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:53.833058 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:10:58.655427 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:10:58.655388 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:10:58.786798 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.786752 4304 generic.go:356] "Generic (PLEG): container finished" podID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerID="98e39dfbcbc9bb032f4dce30db690a191eba9e43c5ad40eff86f437086c664a8" exitCode=0 Nov 03 01:10:58.786922 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.786804 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" event={"ID":"e70c8d9a-48b8-48f2-b985-001e2448f741","Type":"ContainerDied","Data":"98e39dfbcbc9bb032f4dce30db690a191eba9e43c5ad40eff86f437086c664a8"} Nov 03 01:10:58.786922 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.786874 4304 scope.go:122] "RemoveContainer" containerID="866f977aa251107cea02af898260fc7dd824cc29c85825b97d984c955832fca1" Nov 03 01:10:58.930361 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.930341 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:10:58.957562 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.957534 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-kube-api-access-6b7sr\" (UniqueName: \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-kube-api-access-6b7sr\") pod \"e70c8d9a-48b8-48f2-b985-001e2448f741\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " Nov 03 01:10:58.957687 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.957582 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-data\") pod \"e70c8d9a-48b8-48f2-b985-001e2448f741\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " Nov 03 01:10:58.957687 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.957623 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-envoy\") pod \"e70c8d9a-48b8-48f2-b985-001e2448f741\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " Nov 03 01:10:58.957819 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.957712 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e70c8d9a-48b8-48f2-b985-001e2448f741-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70c8d9a-48b8-48f2-b985-001e2448f741-istiod-ca-cert\") pod \"e70c8d9a-48b8-48f2-b985-001e2448f741\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " Nov 03 01:10:58.957819 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.957761 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-credential-socket\") pod \"e70c8d9a-48b8-48f2-b985-001e2448f741\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " Nov 03 01:10:58.957819 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.957807 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-socket\") pod \"e70c8d9a-48b8-48f2-b985-001e2448f741\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " Nov 03 01:10:58.957975 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.957881 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-token\" (UniqueName: \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-token\") pod \"e70c8d9a-48b8-48f2-b985-001e2448f741\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " Nov 03 01:10:58.957975 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.957921 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-certs\") pod \"e70c8d9a-48b8-48f2-b985-001e2448f741\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " Nov 03 01:10:58.957975 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.957932 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-data" pod "e70c8d9a-48b8-48f2-b985-001e2448f741" (UID: "e70c8d9a-48b8-48f2-b985-001e2448f741"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:58.958179 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.957987 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-podinfo\") pod \"e70c8d9a-48b8-48f2-b985-001e2448f741\" (UID: \"e70c8d9a-48b8-48f2-b985-001e2448f741\") " Nov 03 01:10:58.958179 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.958130 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70c8d9a-48b8-48f2-b985-001e2448f741-istiod-ca-cert" pod "e70c8d9a-48b8-48f2-b985-001e2448f741" (UID: "e70c8d9a-48b8-48f2-b985-001e2448f741"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:58.958283 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.958172 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-certs" pod "e70c8d9a-48b8-48f2-b985-001e2448f741" (UID: "e70c8d9a-48b8-48f2-b985-001e2448f741"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:58.958283 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.958206 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-socket" pod "e70c8d9a-48b8-48f2-b985-001e2448f741" (UID: "e70c8d9a-48b8-48f2-b985-001e2448f741"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:58.958544 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.958519 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-credential-socket" pod "e70c8d9a-48b8-48f2-b985-001e2448f741" (UID: "e70c8d9a-48b8-48f2-b985-001e2448f741"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:58.958618 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.958584 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:58.958618 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.958604 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70c8d9a-48b8-48f2-b985-001e2448f741-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:58.958618 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.958617 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:58.958794 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.958630 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:58.960573 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.960547 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-kube-api-access-6b7sr" pod "e70c8d9a-48b8-48f2-b985-001e2448f741" (UID: "e70c8d9a-48b8-48f2-b985-001e2448f741"). InnerVolumeSpecName "kube-api-access-6b7sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:58.960682 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.960594 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-token" pod "e70c8d9a-48b8-48f2-b985-001e2448f741" (UID: "e70c8d9a-48b8-48f2-b985-001e2448f741"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:58.960743 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.960688 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-envoy" pod "e70c8d9a-48b8-48f2-b985-001e2448f741" (UID: "e70c8d9a-48b8-48f2-b985-001e2448f741"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:58.960850 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:58.960828 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-podinfo" pod "e70c8d9a-48b8-48f2-b985-001e2448f741" (UID: "e70c8d9a-48b8-48f2-b985-001e2448f741"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:59.059877 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:59.059851 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b7sr\" (UniqueName: \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-kube-api-access-6b7sr\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:59.059877 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:59.059877 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:59.060016 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:59.059887 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70c8d9a-48b8-48f2-b985-001e2448f741-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:59.060016 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:59.059895 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:59.060016 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:59.059904 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70c8d9a-48b8-48f2-b985-001e2448f741-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:59.794517 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:59.794485 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" Nov 03 01:10:59.795015 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:59.794520 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8" event={"ID":"e70c8d9a-48b8-48f2-b985-001e2448f741","Type":"ContainerDied","Data":"dcd218bcc9048bdbe2223fbf5fbb876526428ab7fea134153f010c9580f8d42d"} Nov 03 01:10:59.795015 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:10:59.794591 4304 scope.go:122] "RemoveContainer" containerID="98e39dfbcbc9bb032f4dce30db690a191eba9e43c5ad40eff86f437086c664a8" Nov 03 01:11:00.007693 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:11:00.007666 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8"] Nov 03 01:11:00.013732 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:11:00.013709 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4db6f5-openshift-default-888874649-whqt8"] Nov 03 01:11:00.829363 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:11:00.829331 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" path="/var/lib/kubelet/pods/e70c8d9a-48b8-48f2-b985-001e2448f741/volumes" Nov 03 01:11:28.654420 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:11:28.654383 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:11:58.655193 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:11:58.655156 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:12:28.654971 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:12:28.654935 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:12:58.653667 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:12:58.653624 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:13:28.654548 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:13:28.654514 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:13:58.653968 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:13:58.653890 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:14:28.655167 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:14:28.655126 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:14:28.994559 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.994489 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85"] Nov 03 01:14:28.995017 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995003 4304 cpu_manager.go:414] "containerMap: removing container" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" Nov 03 01:14:28.995068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995018 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" Nov 03 01:14:28.995068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995028 4304 cpu_manager.go:414] "containerMap: removing container" podUID="6e863b69-239a-4f2f-974e-18dddebcdd1b" containerName="maas-api" Nov 03 01:14:28.995068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995034 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e863b69-239a-4f2f-974e-18dddebcdd1b" containerName="maas-api" Nov 03 01:14:28.995068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995044 4304 cpu_manager.go:414] "containerMap: removing container" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" Nov 03 01:14:28.995068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995048 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" Nov 03 01:14:28.995068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995060 4304 cpu_manager.go:414] "containerMap: removing container" podUID="032fe307-1e47-4c10-885e-10027dec0da7" containerName="payload-pre-processing" Nov 03 01:14:28.995068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995066 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032fe307-1e47-4c10-885e-10027dec0da7" containerName="payload-pre-processing" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995084 4304 cpu_manager.go:414] "containerMap: removing container" podUID="82f60f9c-168b-4f7e-b15a-4deee10e3eb2" containerName="payload-processing" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995089 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f60f9c-168b-4f7e-b15a-4deee10e3eb2" containerName="payload-processing" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995095 4304 cpu_manager.go:414] "containerMap: removing container" podUID="6bd5b850-54fd-4bed-aefa-c0b60f902c44" containerName="maas-api" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995099 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd5b850-54fd-4bed-aefa-c0b60f902c44" containerName="maas-api" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995105 4304 cpu_manager.go:414] "containerMap: removing container" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995109 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995115 4304 cpu_manager.go:414] "containerMap: removing container" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995119 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995198 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f60f9c-168b-4f7e-b15a-4deee10e3eb2" containerName="payload-processing" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995203 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd5b850-54fd-4bed-aefa-c0b60f902c44" containerName="maas-api" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995210 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995214 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995222 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d949865-b1a9-4c64-909b-5d617c81496c" containerName="istio-proxy" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995227 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e863b69-239a-4f2f-974e-18dddebcdd1b" containerName="maas-api" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995232 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70c8d9a-48b8-48f2-b985-001e2448f741" containerName="istio-proxy" Nov 03 01:14:28.995297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.995238 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="032fe307-1e47-4c10-885e-10027dec0da7" containerName="payload-pre-processing" Nov 03 01:14:28.998486 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:28.998468 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.097793 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.097744 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tls-certs\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.097938 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.097872 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-model-cache\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.097987 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.097939 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-dshm\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.097987 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.097972 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-home\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.098055 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.097995 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tmp-dir\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.098144 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.098131 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf66s\" (UniqueName: \"kubernetes.io/projected/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-kube-api-access-kf66s\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.101295 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.101270 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fce94b\"/\"openshift-service-ca.crt\"" Nov 03 01:14:29.101501 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.101484 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fce94b\"/\"rate-test-model-fce94b-kserve-self-signed-certs\"" Nov 03 01:14:29.101554 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.101498 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fce94b\"/\"kube-root-ca.crt\"" Nov 03 01:14:29.101554 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.101525 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fce94b\"/\"default-dockercfg-4mz2z\"" Nov 03 01:14:29.104681 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.104664 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85"] Nov 03 01:14:29.198994 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.198967 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-home\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.199179 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.199004 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tmp-dir\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.199179 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.199094 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf66s\" (UniqueName: \"kubernetes.io/projected/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-kube-api-access-kf66s\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.199179 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.199145 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tls-certs\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.199304 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.199217 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-model-cache\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.199304 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.199258 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-dshm\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.199394 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.199374 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-home\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.199436 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.199418 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tmp-dir\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.199511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.199496 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-model-cache\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.201574 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.201544 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-dshm\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.201822 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.201807 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tls-certs\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.255957 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.255903 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf66s\" (UniqueName: \"kubernetes.io/projected/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-kube-api-access-kf66s\") pod \"rate-test-model-fce94b-kserve-7f65496797-jlw85\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.310891 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.310867 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:29.685554 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:29.685528 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85"] Nov 03 01:14:29.687180 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:14:29.687150 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30838ca2_820a_4d11_8ce5_5dfd2d579bf8.slice/crio-7a398577941d6ea5e88cdc227d5154fa0dce94f68fa8d2154c2f08142d224a0e WatchSource:0}: Error finding container 7a398577941d6ea5e88cdc227d5154fa0dce94f68fa8d2154c2f08142d224a0e: Status 404 returned error can't find the container with id 7a398577941d6ea5e88cdc227d5154fa0dce94f68fa8d2154c2f08142d224a0e Nov 03 01:14:30.146402 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:30.146368 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" event={"ID":"30838ca2-820a-4d11-8ce5-5dfd2d579bf8","Type":"ContainerStarted","Data":"7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74"} Nov 03 01:14:30.146559 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:30.146416 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" event={"ID":"30838ca2-820a-4d11-8ce5-5dfd2d579bf8","Type":"ContainerStarted","Data":"7a398577941d6ea5e88cdc227d5154fa0dce94f68fa8d2154c2f08142d224a0e"} Nov 03 01:14:30.265258 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:30.265209 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" podStartSLOduration=2.265191182 podStartE2EDuration="2.265191182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:30.262435516 +0000 UTC m=+4153.545993189" watchObservedRunningTime="2026-11-03 01:14:30.265191182 +0000 UTC m=+4153.548748855" Nov 03 01:14:30.311926 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:30.311904 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:39.311764 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:39.311732 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:39.323993 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:39.323968 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:40.220804 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:40.220745 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:14:52.327541 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.327514 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk"] Nov 03 01:14:52.331649 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.331634 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.434156 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.434124 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-20f0a3\"/\"kube-root-ca.crt\"" Nov 03 01:14:52.434380 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.434174 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-20f0a3\"/\"openshift-service-ca.crt\"" Nov 03 01:14:52.434594 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.434576 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-20f0a3\"/\"rate-test-model-20f0a3-kserve-self-signed-certs\"" Nov 03 01:14:52.434673 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.434661 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-20f0a3\"/\"default-dockercfg-twj22\"" Nov 03 01:14:52.437291 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.437270 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk"] Nov 03 01:14:52.452293 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.452269 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-dshm\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.452360 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.452306 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-model-cache\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.452360 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.452341 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84pzv\" (UniqueName: \"kubernetes.io/projected/08ce4326-4703-4061-aec4-17b4c6567785-kube-api-access-84pzv\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.452497 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.452482 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-home\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.452539 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.452524 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-tmp-dir\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.452574 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.452544 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ce4326-4703-4061-aec4-17b4c6567785-tls-certs\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.554146 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.554114 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-tmp-dir\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.554146 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.554150 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ce4326-4703-4061-aec4-17b4c6567785-tls-certs\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.554373 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.554245 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-dshm\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.554373 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.554266 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-model-cache\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.554373 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.554289 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84pzv\" (UniqueName: \"kubernetes.io/projected/08ce4326-4703-4061-aec4-17b4c6567785-kube-api-access-84pzv\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.554373 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.554341 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-home\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.554578 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.554525 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-tmp-dir\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.554723 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.554699 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-home\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.554723 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.554716 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-model-cache\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.556668 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.556644 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-dshm\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.556927 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.556912 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ce4326-4703-4061-aec4-17b4c6567785-tls-certs\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.610441 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.610379 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84pzv\" (UniqueName: \"kubernetes.io/projected/08ce4326-4703-4061-aec4-17b4c6567785-kube-api-access-84pzv\") pod \"rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:52.643575 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:52.643551 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:53.019732 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:53.019709 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk"] Nov 03 01:14:53.020695 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:14:53.020670 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08ce4326_4703_4061_aec4_17b4c6567785.slice/crio-a77ef79051754a80da750f5a310040ef6db64ce4b1cd5cb22838da3d9a7c0eeb WatchSource:0}: Error finding container a77ef79051754a80da750f5a310040ef6db64ce4b1cd5cb22838da3d9a7c0eeb: Status 404 returned error can't find the container with id a77ef79051754a80da750f5a310040ef6db64ce4b1cd5cb22838da3d9a7c0eeb Nov 03 01:14:53.293829 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:53.293726 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" event={"ID":"08ce4326-4703-4061-aec4-17b4c6567785","Type":"ContainerStarted","Data":"738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101"} Nov 03 01:14:53.293829 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:53.293791 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" event={"ID":"08ce4326-4703-4061-aec4-17b4c6567785","Type":"ContainerStarted","Data":"a77ef79051754a80da750f5a310040ef6db64ce4b1cd5cb22838da3d9a7c0eeb"} Nov 03 01:14:53.412212 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:53.412162 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" podStartSLOduration=1.412149219 podStartE2EDuration="1.412149219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:53.408988606 +0000 UTC m=+4176.692546281" watchObservedRunningTime="2026-11-03 01:14:53.412149219 +0000 UTC m=+4176.695706893" Nov 03 01:14:53.644614 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:14:53.644588 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:14:58.654897 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:14:58.654856 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:15:00.254957 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:00.254884 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw"] Nov 03 01:15:00.258829 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:00.258814 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" Nov 03 01:15:00.338826 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:00.338795 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:15:00.436350 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:00.436320 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhs27\" (UniqueName: \"kubernetes.io/projected/33faec4b-6d26-45f9-b8bb-104d131d8040-kube-api-access-jhs27\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw\" (UID: \"33faec4b-6d26-45f9-b8bb-104d131d8040\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" Nov 03 01:15:00.436514 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:00.436374 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33faec4b-6d26-45f9-b8bb-104d131d8040-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw\" (UID: \"33faec4b-6d26-45f9-b8bb-104d131d8040\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" Nov 03 01:15:00.450649 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:00.450601 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw"] Nov 03 01:15:00.537865 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:00.537770 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhs27\" (UniqueName: \"kubernetes.io/projected/33faec4b-6d26-45f9-b8bb-104d131d8040-kube-api-access-jhs27\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw\" (UID: \"33faec4b-6d26-45f9-b8bb-104d131d8040\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" Nov 03 01:15:00.537865 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:00.537858 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33faec4b-6d26-45f9-b8bb-104d131d8040-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw\" (UID: \"33faec4b-6d26-45f9-b8bb-104d131d8040\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" Nov 03 01:15:00.538432 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:00.538413 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33faec4b-6d26-45f9-b8bb-104d131d8040-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw\" (UID: \"33faec4b-6d26-45f9-b8bb-104d131d8040\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" Nov 03 01:15:00.603590 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:00.603557 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhs27\" (UniqueName: \"kubernetes.io/projected/33faec4b-6d26-45f9-b8bb-104d131d8040-kube-api-access-jhs27\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw\" (UID: \"33faec4b-6d26-45f9-b8bb-104d131d8040\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" Nov 03 01:15:00.873200 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:00.873131 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" Nov 03 01:15:01.250653 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:01.250630 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw"] Nov 03 01:15:01.252188 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:15:01.252163 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33faec4b_6d26_45f9_b8bb_104d131d8040.slice/crio-7104c4de097d4f94bb3fdc26f8392804fdb67d6868e666a5fe4e2fc0a129feef WatchSource:0}: Error finding container 7104c4de097d4f94bb3fdc26f8392804fdb67d6868e666a5fe4e2fc0a129feef: Status 404 returned error can't find the container with id 7104c4de097d4f94bb3fdc26f8392804fdb67d6868e666a5fe4e2fc0a129feef Nov 03 01:15:01.346311 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:01.346281 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" event={"ID":"33faec4b-6d26-45f9-b8bb-104d131d8040","Type":"ContainerStarted","Data":"7104c4de097d4f94bb3fdc26f8392804fdb67d6868e666a5fe4e2fc0a129feef"} Nov 03 01:15:02.354740 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:02.354657 4304 generic.go:356] "Generic (PLEG): container finished" podID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerID="dac9469a2f28435f3312efd9fd634884d7feeaf549a43422ee5b820ba3312e97" exitCode=60 Nov 03 01:15:02.354740 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:02.354721 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" event={"ID":"33faec4b-6d26-45f9-b8bb-104d131d8040","Type":"ContainerDied","Data":"dac9469a2f28435f3312efd9fd634884d7feeaf549a43422ee5b820ba3312e97"} Nov 03 01:15:02.355164 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:02.355021 4304 scope.go:122] "RemoveContainer" containerID="dac9469a2f28435f3312efd9fd634884d7feeaf549a43422ee5b820ba3312e97" Nov 03 01:15:02.643939 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:02.643917 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:15:02.657471 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:02.657452 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:15:03.365737 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:03.365702 4304 generic.go:356] "Generic (PLEG): container finished" podID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerID="0971d2a4444038cff1b23d0443edd2304300878e1fb7eadf9815eb1bc7cced0a" exitCode=60 Nov 03 01:15:03.366140 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:03.365790 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" event={"ID":"33faec4b-6d26-45f9-b8bb-104d131d8040","Type":"ContainerDied","Data":"0971d2a4444038cff1b23d0443edd2304300878e1fb7eadf9815eb1bc7cced0a"} Nov 03 01:15:03.366140 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:03.365851 4304 scope.go:122] "RemoveContainer" containerID="dac9469a2f28435f3312efd9fd634884d7feeaf549a43422ee5b820ba3312e97" Nov 03 01:15:03.366140 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:03.366076 4304 scope.go:122] "RemoveContainer" containerID="0971d2a4444038cff1b23d0443edd2304300878e1fb7eadf9815eb1bc7cced0a" Nov 03 01:15:03.366430 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:15:03.366391 4304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw_odh-ai-gateway-infra(33faec4b-6d26-45f9-b8bb-104d131d8040)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" Nov 03 01:15:03.379550 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:03.379527 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:15:04.374065 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:04.374026 4304 scope.go:122] "RemoveContainer" containerID="0971d2a4444038cff1b23d0443edd2304300878e1fb7eadf9815eb1bc7cced0a" Nov 03 01:15:04.374515 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:15:04.374304 4304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw_odh-ai-gateway-infra(33faec4b-6d26-45f9-b8bb-104d131d8040)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" Nov 03 01:15:16.825766 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:16.825735 4304 scope.go:122] "RemoveContainer" containerID="0971d2a4444038cff1b23d0443edd2304300878e1fb7eadf9815eb1bc7cced0a" Nov 03 01:15:17.454152 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:17.454116 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" event={"ID":"33faec4b-6d26-45f9-b8bb-104d131d8040","Type":"ContainerStarted","Data":"d40bef1dc80eb81b2e90b17000fe13f02bf4f2be0535cf118cdae00c3ed324c9"} Nov 03 01:15:17.569450 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:17.569398 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" podStartSLOduration=17.569384133 podStartE2EDuration="17.569384133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:17.568440563 +0000 UTC m=+4200.851998236" watchObservedRunningTime="2026-11-03 01:15:17.569384133 +0000 UTC m=+4200.852941815" Nov 03 01:15:18.461643 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:18.461612 4304 generic.go:356] "Generic (PLEG): container finished" podID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerID="d40bef1dc80eb81b2e90b17000fe13f02bf4f2be0535cf118cdae00c3ed324c9" exitCode=60 Nov 03 01:15:18.462065 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:18.461683 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" event={"ID":"33faec4b-6d26-45f9-b8bb-104d131d8040","Type":"ContainerDied","Data":"d40bef1dc80eb81b2e90b17000fe13f02bf4f2be0535cf118cdae00c3ed324c9"} Nov 03 01:15:18.462065 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:18.461741 4304 scope.go:122] "RemoveContainer" containerID="0971d2a4444038cff1b23d0443edd2304300878e1fb7eadf9815eb1bc7cced0a" Nov 03 01:15:18.462065 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:18.462050 4304 scope.go:122] "RemoveContainer" containerID="d40bef1dc80eb81b2e90b17000fe13f02bf4f2be0535cf118cdae00c3ed324c9" Nov 03 01:15:18.462391 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:15:18.462371 4304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw_odh-ai-gateway-infra(33faec4b-6d26-45f9-b8bb-104d131d8040)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" Nov 03 01:15:18.587860 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:18.587823 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw"] Nov 03 01:15:19.619135 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:19.619110 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" Nov 03 01:15:19.735934 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:19.735833 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33faec4b-6d26-45f9-b8bb-104d131d8040-kube-api-access-jhs27\" (UniqueName: \"kubernetes.io/projected/33faec4b-6d26-45f9-b8bb-104d131d8040-kube-api-access-jhs27\") pod \"33faec4b-6d26-45f9-b8bb-104d131d8040\" (UID: \"33faec4b-6d26-45f9-b8bb-104d131d8040\") " Nov 03 01:15:19.735934 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:19.735890 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33faec4b-6d26-45f9-b8bb-104d131d8040-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33faec4b-6d26-45f9-b8bb-104d131d8040-maas-api-service-ca\") pod \"33faec4b-6d26-45f9-b8bb-104d131d8040\" (UID: \"33faec4b-6d26-45f9-b8bb-104d131d8040\") " Nov 03 01:15:19.737548 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:19.736825 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33faec4b-6d26-45f9-b8bb-104d131d8040-maas-api-service-ca" pod "33faec4b-6d26-45f9-b8bb-104d131d8040" (UID: "33faec4b-6d26-45f9-b8bb-104d131d8040"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:19.739096 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:19.739066 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33faec4b-6d26-45f9-b8bb-104d131d8040-kube-api-access-jhs27" pod "33faec4b-6d26-45f9-b8bb-104d131d8040" (UID: "33faec4b-6d26-45f9-b8bb-104d131d8040"). InnerVolumeSpecName "kube-api-access-jhs27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:19.837835 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:19.837803 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhs27\" (UniqueName: \"kubernetes.io/projected/33faec4b-6d26-45f9-b8bb-104d131d8040-kube-api-access-jhs27\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.837835 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:19.837829 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33faec4b-6d26-45f9-b8bb-104d131d8040-maas-api-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:20.479263 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:20.479233 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" event={"ID":"33faec4b-6d26-45f9-b8bb-104d131d8040","Type":"ContainerDied","Data":"7104c4de097d4f94bb3fdc26f8392804fdb67d6868e666a5fe4e2fc0a129feef"} Nov 03 01:15:20.479475 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:20.479269 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw" Nov 03 01:15:20.479475 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:20.479292 4304 scope.go:122] "RemoveContainer" containerID="d40bef1dc80eb81b2e90b17000fe13f02bf4f2be0535cf118cdae00c3ed324c9" Nov 03 01:15:20.644791 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:20.644741 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw"] Nov 03 01:15:20.694207 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:20.694174 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894475-szbfw"] Nov 03 01:15:20.830366 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:15:20.830290 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" path="/var/lib/kubelet/pods/33faec4b-6d26-45f9-b8bb-104d131d8040/volumes" Nov 03 01:15:28.655688 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:15:28.655647 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:15:58.654850 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:15:58.654808 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:16:06.713018 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:06.712988 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm"] Nov 03 01:16:06.821487 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:06.821457 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6"] Nov 03 01:16:09.170442 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:09.170410 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx"] Nov 03 01:16:11.713864 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:11.713798 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" podUID="4d155e9a-254a-4bb0-a535-1c237d7e1f8c" containerName="payload-processing" containerID="cri-o://9ba6608df492dc5dbe1b40cfa7c7a2a93bedfc56fc231b341aec9b14fe7218b7" gracePeriod=25 Nov 03 01:16:11.821909 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:11.821874 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" podUID="010ce23a-6dab-4d92-a5ce-acc072a62fd0" containerName="payload-pre-processing" containerID="cri-o://7ae23cd759b45dcd8fd30ede9e72ab88d4e33e2b3be363440822acfffad3af84" gracePeriod=25 Nov 03 01:16:12.896024 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:12.895985 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" podUID="4d155e9a-254a-4bb0-a535-1c237d7e1f8c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:16:12.896417 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:12.896005 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" podUID="010ce23a-6dab-4d92-a5ce-acc072a62fd0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:16:14.171699 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:14.171659 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" podUID="7c9c0826-a79b-44de-ab30-37a80157c40d" containerName="payload-processing" containerID="cri-o://f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352" gracePeriod=25 Nov 03 01:16:16.226447 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:16.226411 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" podUID="7c9c0826-a79b-44de-ab30-37a80157c40d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:16:22.895552 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:22.895514 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" podUID="4d155e9a-254a-4bb0-a535-1c237d7e1f8c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:16:22.895944 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:22.895552 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" podUID="010ce23a-6dab-4d92-a5ce-acc072a62fd0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:16:26.226203 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:26.226171 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" podUID="7c9c0826-a79b-44de-ab30-37a80157c40d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:16:28.655725 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:16:28.655621 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:16:31.968296 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:31.968257 4304 generic.go:356] "Generic (PLEG): container finished" podID="010ce23a-6dab-4d92-a5ce-acc072a62fd0" containerID="7ae23cd759b45dcd8fd30ede9e72ab88d4e33e2b3be363440822acfffad3af84" exitCode=0 Nov 03 01:16:31.968595 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:31.968475 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" event={"ID":"010ce23a-6dab-4d92-a5ce-acc072a62fd0","Type":"ContainerDied","Data":"7ae23cd759b45dcd8fd30ede9e72ab88d4e33e2b3be363440822acfffad3af84"} Nov 03 01:16:31.970158 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:31.970137 4304 generic.go:356] "Generic (PLEG): container finished" podID="4d155e9a-254a-4bb0-a535-1c237d7e1f8c" containerID="9ba6608df492dc5dbe1b40cfa7c7a2a93bedfc56fc231b341aec9b14fe7218b7" exitCode=0 Nov 03 01:16:31.970238 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:31.970185 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" event={"ID":"4d155e9a-254a-4bb0-a535-1c237d7e1f8c","Type":"ContainerDied","Data":"9ba6608df492dc5dbe1b40cfa7c7a2a93bedfc56fc231b341aec9b14fe7218b7"} Nov 03 01:16:31.990621 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:31.990595 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 01:16:32.107647 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.107615 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 01:16:32.152212 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.152178 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-plugins-config-volume\") pod \"4d155e9a-254a-4bb0-a535-1c237d7e1f8c\" (UID: \"4d155e9a-254a-4bb0-a535-1c237d7e1f8c\") " Nov 03 01:16:32.152389 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.152241 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-kube-api-access-hmlm5\" (UniqueName: \"kubernetes.io/projected/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-kube-api-access-hmlm5\") pod \"4d155e9a-254a-4bb0-a535-1c237d7e1f8c\" (UID: \"4d155e9a-254a-4bb0-a535-1c237d7e1f8c\") " Nov 03 01:16:32.152586 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.152554 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-plugins-config-volume" pod "4d155e9a-254a-4bb0-a535-1c237d7e1f8c" (UID: "4d155e9a-254a-4bb0-a535-1c237d7e1f8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:32.152730 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.152712 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:32.154446 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.154426 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-kube-api-access-hmlm5" pod "4d155e9a-254a-4bb0-a535-1c237d7e1f8c" (UID: "4d155e9a-254a-4bb0-a535-1c237d7e1f8c"). InnerVolumeSpecName "kube-api-access-hmlm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:32.253653 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.253572 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010ce23a-6dab-4d92-a5ce-acc072a62fd0-kube-api-access-l8x9s\" (UniqueName: \"kubernetes.io/projected/010ce23a-6dab-4d92-a5ce-acc072a62fd0-kube-api-access-l8x9s\") pod \"010ce23a-6dab-4d92-a5ce-acc072a62fd0\" (UID: \"010ce23a-6dab-4d92-a5ce-acc072a62fd0\") " Nov 03 01:16:32.253834 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.253725 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010ce23a-6dab-4d92-a5ce-acc072a62fd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010ce23a-6dab-4d92-a5ce-acc072a62fd0-plugins-config-volume\") pod \"010ce23a-6dab-4d92-a5ce-acc072a62fd0\" (UID: \"010ce23a-6dab-4d92-a5ce-acc072a62fd0\") " Nov 03 01:16:32.254079 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.254058 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmlm5\" (UniqueName: \"kubernetes.io/projected/4d155e9a-254a-4bb0-a535-1c237d7e1f8c-kube-api-access-hmlm5\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:32.254174 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.254115 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010ce23a-6dab-4d92-a5ce-acc072a62fd0-plugins-config-volume" pod "010ce23a-6dab-4d92-a5ce-acc072a62fd0" (UID: "010ce23a-6dab-4d92-a5ce-acc072a62fd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:32.255883 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.255861 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010ce23a-6dab-4d92-a5ce-acc072a62fd0-kube-api-access-l8x9s" pod "010ce23a-6dab-4d92-a5ce-acc072a62fd0" (UID: "010ce23a-6dab-4d92-a5ce-acc072a62fd0"). InnerVolumeSpecName "kube-api-access-l8x9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:32.354816 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.354754 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010ce23a-6dab-4d92-a5ce-acc072a62fd0-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:32.354816 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.354812 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8x9s\" (UniqueName: \"kubernetes.io/projected/010ce23a-6dab-4d92-a5ce-acc072a62fd0-kube-api-access-l8x9s\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:32.466453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.466419 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5"] Nov 03 01:16:32.467095 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467074 4304 cpu_manager.go:414] "containerMap: removing container" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerName="cleanup" Nov 03 01:16:32.467095 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467095 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerName="cleanup" Nov 03 01:16:32.467283 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467121 4304 cpu_manager.go:414] "containerMap: removing container" podUID="4d155e9a-254a-4bb0-a535-1c237d7e1f8c" containerName="payload-processing" Nov 03 01:16:32.467283 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467128 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d155e9a-254a-4bb0-a535-1c237d7e1f8c" containerName="payload-processing" Nov 03 01:16:32.467283 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467145 4304 cpu_manager.go:414] "containerMap: removing container" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerName="cleanup" Nov 03 01:16:32.467283 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467153 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerName="cleanup" Nov 03 01:16:32.467283 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467176 4304 cpu_manager.go:414] "containerMap: removing container" podUID="010ce23a-6dab-4d92-a5ce-acc072a62fd0" containerName="payload-pre-processing" Nov 03 01:16:32.467283 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467183 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010ce23a-6dab-4d92-a5ce-acc072a62fd0" containerName="payload-pre-processing" Nov 03 01:16:32.467576 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467308 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="010ce23a-6dab-4d92-a5ce-acc072a62fd0" containerName="payload-pre-processing" Nov 03 01:16:32.467576 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467325 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerName="cleanup" Nov 03 01:16:32.467576 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467336 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerName="cleanup" Nov 03 01:16:32.467576 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467344 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d155e9a-254a-4bb0-a535-1c237d7e1f8c" containerName="payload-processing" Nov 03 01:16:32.467576 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.467356 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerName="cleanup" Nov 03 01:16:32.470912 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.470891 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.522882 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.522811 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ebeeb9-openshift-default-dockercfg-pt5k5\"" Nov 03 01:16:32.530053 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.530030 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5"] Nov 03 01:16:32.658233 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.658192 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.658396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.658241 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.658443 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.658388 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c060e707-47c0-4e79-b386-80d66b4f33dc-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.658443 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.658416 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.658518 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.658444 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.658564 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.658532 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vsz6\" (UniqueName: \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-kube-api-access-5vsz6\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.658599 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.658562 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.658640 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.658595 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.658675 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.658642 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.759271 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759237 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c060e707-47c0-4e79-b386-80d66b4f33dc-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.759409 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759275 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.759409 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759310 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.759409 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759351 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vsz6\" (UniqueName: \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-kube-api-access-5vsz6\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.759409 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759381 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.759640 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759419 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.759640 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759488 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.759805 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759757 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.759928 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759821 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.759928 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759846 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.760052 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759879 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.760052 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.759957 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.760174 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.760049 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c060e707-47c0-4e79-b386-80d66b4f33dc-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.760231 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.760195 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.761989 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.761968 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.762302 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.762286 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.841371 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.841348 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vsz6\" (UniqueName: \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-kube-api-access-5vsz6\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.841470 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.841372 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:32.978558 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.978530 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" Nov 03 01:16:32.978978 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.978560 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6" event={"ID":"010ce23a-6dab-4d92-a5ce-acc072a62fd0","Type":"ContainerDied","Data":"cefc693f03b0de4c1b630735c09051897117aff1309a36d4b7b51b38164212f1"} Nov 03 01:16:32.978978 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.978618 4304 scope.go:122] "RemoveContainer" containerID="7ae23cd759b45dcd8fd30ede9e72ab88d4e33e2b3be363440822acfffad3af84" Nov 03 01:16:32.980240 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.980218 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" Nov 03 01:16:32.980240 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.980225 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm" event={"ID":"4d155e9a-254a-4bb0-a535-1c237d7e1f8c","Type":"ContainerDied","Data":"a85873914764229b05fc0b62848c7b9663473f7a2884d7fe59f20352e9a86a1e"} Nov 03 01:16:32.991379 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:32.991363 4304 scope.go:122] "RemoveContainer" containerID="9ba6608df492dc5dbe1b40cfa7c7a2a93bedfc56fc231b341aec9b14fe7218b7" Nov 03 01:16:33.084490 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:33.084464 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:33.143534 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:33.143477 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm"] Nov 03 01:16:33.157430 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:33.157404 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ebeeb9-768ddc954c-z2tdm"] Nov 03 01:16:33.270094 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:33.270067 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6"] Nov 03 01:16:33.273937 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:33.273914 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ebeeb9-999964c5f-8jvk6"] Nov 03 01:16:33.510572 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:33.510550 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5"] Nov 03 01:16:33.511949 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:16:33.511913 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc060e707_47c0_4e79_b386_80d66b4f33dc.slice/crio-c6d11756336f1f908312b04cf75d0074b3e08f98ce0afccae9e96a87edf57c4d WatchSource:0}: Error finding container c6d11756336f1f908312b04cf75d0074b3e08f98ce0afccae9e96a87edf57c4d: Status 404 returned error can't find the container with id c6d11756336f1f908312b04cf75d0074b3e08f98ce0afccae9e96a87edf57c4d Nov 03 01:16:33.514604 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:33.514574 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:16:33.514683 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:33.514637 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:16:33.514683 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:33.514664 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:16:33.992361 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:33.992326 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" event={"ID":"c060e707-47c0-4e79-b386-80d66b4f33dc","Type":"ContainerStarted","Data":"d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69"} Nov 03 01:16:33.992760 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:33.992384 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" event={"ID":"c060e707-47c0-4e79-b386-80d66b4f33dc","Type":"ContainerStarted","Data":"c6d11756336f1f908312b04cf75d0074b3e08f98ce0afccae9e96a87edf57c4d"} Nov 03 01:16:34.085726 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.085689 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:34.090209 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.090176 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:16:34.090389 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.090254 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 01:16:34.108643 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.108589 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" podStartSLOduration=2.108576076 podStartE2EDuration="2.108576076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:34.108161771 +0000 UTC m=+4277.391719459" watchObservedRunningTime="2026-11-03 01:16:34.108576076 +0000 UTC m=+4277.392133818" Nov 03 01:16:34.483754 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.483729 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 01:16:34.585675 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.585580 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9c0826-a79b-44de-ab30-37a80157c40d-kube-api-access-zk9x8\" (UniqueName: \"kubernetes.io/projected/7c9c0826-a79b-44de-ab30-37a80157c40d-kube-api-access-zk9x8\") pod \"7c9c0826-a79b-44de-ab30-37a80157c40d\" (UID: \"7c9c0826-a79b-44de-ab30-37a80157c40d\") " Nov 03 01:16:34.585675 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.585669 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9c0826-a79b-44de-ab30-37a80157c40d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9c0826-a79b-44de-ab30-37a80157c40d-plugins-config-volume\") pod \"7c9c0826-a79b-44de-ab30-37a80157c40d\" (UID: \"7c9c0826-a79b-44de-ab30-37a80157c40d\") " Nov 03 01:16:34.586065 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.586045 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9c0826-a79b-44de-ab30-37a80157c40d-plugins-config-volume" pod "7c9c0826-a79b-44de-ab30-37a80157c40d" (UID: "7c9c0826-a79b-44de-ab30-37a80157c40d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:34.586311 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.586295 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9c0826-a79b-44de-ab30-37a80157c40d-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:34.587991 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.587967 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9c0826-a79b-44de-ab30-37a80157c40d-kube-api-access-zk9x8" pod "7c9c0826-a79b-44de-ab30-37a80157c40d" (UID: "7c9c0826-a79b-44de-ab30-37a80157c40d"). InnerVolumeSpecName "kube-api-access-zk9x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:34.687796 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.687735 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk9x8\" (UniqueName: \"kubernetes.io/projected/7c9c0826-a79b-44de-ab30-37a80157c40d-kube-api-access-zk9x8\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:34.855094 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.855017 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010ce23a-6dab-4d92-a5ce-acc072a62fd0" path="/var/lib/kubelet/pods/010ce23a-6dab-4d92-a5ce-acc072a62fd0/volumes" Nov 03 01:16:34.855622 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:34.855606 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d155e9a-254a-4bb0-a535-1c237d7e1f8c" path="/var/lib/kubelet/pods/4d155e9a-254a-4bb0-a535-1c237d7e1f8c/volumes" Nov 03 01:16:35.003050 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:35.003011 4304 generic.go:356] "Generic (PLEG): container finished" podID="7c9c0826-a79b-44de-ab30-37a80157c40d" containerID="f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352" exitCode=0 Nov 03 01:16:35.003483 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:35.003065 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" Nov 03 01:16:35.003483 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:35.003096 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" event={"ID":"7c9c0826-a79b-44de-ab30-37a80157c40d","Type":"ContainerDied","Data":"f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352"} Nov 03 01:16:35.003483 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:35.003175 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx" event={"ID":"7c9c0826-a79b-44de-ab30-37a80157c40d","Type":"ContainerDied","Data":"260d409101c094a6da82ce6f31b0cc28890bacf189b3cd180e7f2a385de746f7"} Nov 03 01:16:35.003483 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:35.003202 4304 scope.go:122] "RemoveContainer" containerID="f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352" Nov 03 01:16:35.013395 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:35.013379 4304 scope.go:122] "RemoveContainer" containerID="f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352" Nov 03 01:16:35.013660 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:16:35.013638 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352\": container with ID starting with f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352 not found: ID does not exist" containerID="f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352" Nov 03 01:16:35.013732 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:35.013672 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352"} err="failed to get container status \"f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352\": rpc error: code = NotFound desc = could not find container \"f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352\": container with ID starting with f0b8202661dbaaafdcb1b80140e734b9b9640b0a10cde4a121add917c47e2352 not found: ID does not exist" Nov 03 01:16:35.085460 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:35.085428 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:16:35.085632 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:35.085479 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 01:16:35.167047 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:35.166987 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx"] Nov 03 01:16:35.195390 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:35.195366 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4c8b1-84c7957b85-4zrkx"] Nov 03 01:16:36.085044 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:36.085009 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:16:36.085430 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:36.085080 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 01:16:36.853417 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:36.853388 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9c0826-a79b-44de-ab30-37a80157c40d" path="/var/lib/kubelet/pods/7c9c0826-a79b-44de-ab30-37a80157c40d/volumes" Nov 03 01:16:37.085795 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:37.085747 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:16:37.086180 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:37.085826 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 01:16:38.085580 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:38.085545 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:16:38.085859 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:38.085607 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 01:16:39.085752 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:39.085719 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:16:39.085946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:39.085795 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 01:16:40.084807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:40.084745 4304 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:16:40.085012 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:40.084817 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 01:16:41.089828 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:41.089796 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:41.090230 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:41.090177 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:41.090723 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:16:41.090706 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:16:58.654350 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:16:58.654313 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:17:13.427327 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.427285 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z"] Nov 03 01:17:13.427839 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.427496 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" podUID="4a4c84b0-2ca5-4173-a51a-83116c37bc8e" containerName="maas-api" containerID="cri-o://d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe" gracePeriod=30 Nov 03 01:17:13.708437 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.708414 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 01:17:13.846695 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.844701 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-metrics-certs\") pod \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " Nov 03 01:17:13.846695 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.844834 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-openshift-service-ca\") pod \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " Nov 03 01:17:13.846695 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.844895 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-maas-api-tls\") pod \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " Nov 03 01:17:13.846695 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.845003 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-kube-api-access-qtwrd\" (UniqueName: \"kubernetes.io/projected/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-kube-api-access-qtwrd\") pod \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\" (UID: \"4a4c84b0-2ca5-4173-a51a-83116c37bc8e\") " Nov 03 01:17:13.846695 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.846651 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-openshift-service-ca" pod "4a4c84b0-2ca5-4173-a51a-83116c37bc8e" (UID: "4a4c84b0-2ca5-4173-a51a-83116c37bc8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:13.877081 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.876449 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-kube-api-access-qtwrd" pod "4a4c84b0-2ca5-4173-a51a-83116c37bc8e" (UID: "4a4c84b0-2ca5-4173-a51a-83116c37bc8e"). InnerVolumeSpecName "kube-api-access-qtwrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:13.877081 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.876759 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-maas-api-tls" pod "4a4c84b0-2ca5-4173-a51a-83116c37bc8e" (UID: "4a4c84b0-2ca5-4173-a51a-83116c37bc8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:17:13.877605 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.877569 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-metrics-certs" pod "4a4c84b0-2ca5-4173-a51a-83116c37bc8e" (UID: "4a4c84b0-2ca5-4173-a51a-83116c37bc8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:17:13.946572 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.946476 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:13.946572 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.946508 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:13.946572 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.946518 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtwrd\" (UniqueName: \"kubernetes.io/projected/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-kube-api-access-qtwrd\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:13.946572 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:13.946527 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4c84b0-2ca5-4173-a51a-83116c37bc8e-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:14.293379 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:14.293297 4304 generic.go:356] "Generic (PLEG): container finished" podID="4a4c84b0-2ca5-4173-a51a-83116c37bc8e" containerID="d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe" exitCode=0 Nov 03 01:17:14.293379 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:14.293344 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" Nov 03 01:17:14.293560 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:14.293382 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" event={"ID":"4a4c84b0-2ca5-4173-a51a-83116c37bc8e","Type":"ContainerDied","Data":"d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe"} Nov 03 01:17:14.293560 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:14.293436 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z" event={"ID":"4a4c84b0-2ca5-4173-a51a-83116c37bc8e","Type":"ContainerDied","Data":"2dda8ac69b2003d3f5b10966d186ad7dddb856ac6334fde1242eb057412f451e"} Nov 03 01:17:14.293560 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:14.293462 4304 scope.go:122] "RemoveContainer" containerID="d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe" Nov 03 01:17:14.308733 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:14.308716 4304 scope.go:122] "RemoveContainer" containerID="d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe" Nov 03 01:17:14.309021 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:17:14.309002 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe\": container with ID starting with d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe not found: ID does not exist" containerID="d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe" Nov 03 01:17:14.309085 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:14.309029 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe"} err="failed to get container status \"d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe\": rpc error: code = NotFound desc = could not find container \"d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe\": container with ID starting with d01139497c590a52126be709be12578f1746d14c68468e46fa109f98a2c76bfe not found: ID does not exist" Nov 03 01:17:14.461858 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:14.461829 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z"] Nov 03 01:17:14.464932 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:14.464908 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ebeeb9-d46877567-56k2z"] Nov 03 01:17:14.830359 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:14.830331 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a4c84b0-2ca5-4173-a51a-83116c37bc8e" path="/var/lib/kubelet/pods/4a4c84b0-2ca5-4173-a51a-83116c37bc8e/volumes" Nov 03 01:17:17.155957 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:17.155921 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t"] Nov 03 01:17:17.156419 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:17.156140 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" containerID="cri-o://42e2019fee0da551093eb3a596e7dd130e82b3ad6c6f110480bc0e67dd531cba" gracePeriod=30 Nov 03 01:17:20.492401 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:20.492373 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5"] Nov 03 01:17:20.492815 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:20.492593 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerName="istio-proxy" containerID="cri-o://d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69" gracePeriod=30 Nov 03 01:17:22.351574 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.351542 4304 generic.go:356] "Generic (PLEG): container finished" podID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerID="42e2019fee0da551093eb3a596e7dd130e82b3ad6c6f110480bc0e67dd531cba" exitCode=0 Nov 03 01:17:22.352003 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.351616 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" event={"ID":"dff01bc5-8f42-4d85-af36-90fb2728efe4","Type":"ContainerDied","Data":"42e2019fee0da551093eb3a596e7dd130e82b3ad6c6f110480bc0e67dd531cba"} Nov 03 01:17:22.352003 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.351729 4304 scope.go:122] "RemoveContainer" containerID="86d7a3b8ad60a7bd7136d55c2c93c10312db219c6d62ec891fe14a307cad1e04" Nov 03 01:17:22.428817 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.428737 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 01:17:22.560996 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.560968 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-token\" (UniqueName: \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-token\") pod \"dff01bc5-8f42-4d85-af36-90fb2728efe4\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " Nov 03 01:17:22.561144 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561028 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-envoy\") pod \"dff01bc5-8f42-4d85-af36-90fb2728efe4\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " Nov 03 01:17:22.561144 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561072 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-certs\") pod \"dff01bc5-8f42-4d85-af36-90fb2728efe4\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " Nov 03 01:17:22.561144 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561124 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-podinfo\") pod \"dff01bc5-8f42-4d85-af36-90fb2728efe4\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " Nov 03 01:17:22.561249 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561180 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-credential-socket\") pod \"dff01bc5-8f42-4d85-af36-90fb2728efe4\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " Nov 03 01:17:22.561307 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561278 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-data\") pod \"dff01bc5-8f42-4d85-af36-90fb2728efe4\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " Nov 03 01:17:22.561368 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561317 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-socket\") pod \"dff01bc5-8f42-4d85-af36-90fb2728efe4\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " Nov 03 01:17:22.561368 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561360 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-kube-api-access-sx94n\" (UniqueName: \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-kube-api-access-sx94n\") pod \"dff01bc5-8f42-4d85-af36-90fb2728efe4\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " Nov 03 01:17:22.561485 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561358 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-certs" pod "dff01bc5-8f42-4d85-af36-90fb2728efe4" (UID: "dff01bc5-8f42-4d85-af36-90fb2728efe4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:22.561485 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561388 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff01bc5-8f42-4d85-af36-90fb2728efe4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff01bc5-8f42-4d85-af36-90fb2728efe4-istiod-ca-cert\") pod \"dff01bc5-8f42-4d85-af36-90fb2728efe4\" (UID: \"dff01bc5-8f42-4d85-af36-90fb2728efe4\") " Nov 03 01:17:22.561599 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561505 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-credential-socket" pod "dff01bc5-8f42-4d85-af36-90fb2728efe4" (UID: "dff01bc5-8f42-4d85-af36-90fb2728efe4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:22.561659 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561596 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-socket" pod "dff01bc5-8f42-4d85-af36-90fb2728efe4" (UID: "dff01bc5-8f42-4d85-af36-90fb2728efe4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:22.562009 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.561985 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-data" pod "dff01bc5-8f42-4d85-af36-90fb2728efe4" (UID: "dff01bc5-8f42-4d85-af36-90fb2728efe4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:22.562157 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.562084 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff01bc5-8f42-4d85-af36-90fb2728efe4-istiod-ca-cert" pod "dff01bc5-8f42-4d85-af36-90fb2728efe4" (UID: "dff01bc5-8f42-4d85-af36-90fb2728efe4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:22.562157 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.562001 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:22.562335 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.562169 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:22.562335 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.562185 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:22.563448 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.563426 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-token" pod "dff01bc5-8f42-4d85-af36-90fb2728efe4" (UID: "dff01bc5-8f42-4d85-af36-90fb2728efe4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:22.563552 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.563520 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-kube-api-access-sx94n" pod "dff01bc5-8f42-4d85-af36-90fb2728efe4" (UID: "dff01bc5-8f42-4d85-af36-90fb2728efe4"). InnerVolumeSpecName "kube-api-access-sx94n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:22.563804 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.563763 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-envoy" pod "dff01bc5-8f42-4d85-af36-90fb2728efe4" (UID: "dff01bc5-8f42-4d85-af36-90fb2728efe4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:22.563804 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.563788 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-podinfo" pod "dff01bc5-8f42-4d85-af36-90fb2728efe4" (UID: "dff01bc5-8f42-4d85-af36-90fb2728efe4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:17:22.663172 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.663145 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:22.663172 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.663169 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx94n\" (UniqueName: \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-kube-api-access-sx94n\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:22.663357 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.663179 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff01bc5-8f42-4d85-af36-90fb2728efe4-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:22.663357 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.663190 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:22.663357 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.663206 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:22.663357 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:22.663214 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff01bc5-8f42-4d85-af36-90fb2728efe4-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:23.359387 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:23.359355 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" Nov 03 01:17:23.359812 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:23.359399 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t" event={"ID":"dff01bc5-8f42-4d85-af36-90fb2728efe4","Type":"ContainerDied","Data":"a3f8b841ddc401090515a04c92449e93e71115b5459fed838d3560e956e085ae"} Nov 03 01:17:23.359812 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:23.359453 4304 scope.go:122] "RemoveContainer" containerID="42e2019fee0da551093eb3a596e7dd130e82b3ad6c6f110480bc0e67dd531cba" Nov 03 01:17:23.571627 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:23.571598 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t"] Nov 03 01:17:23.621291 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:23.621225 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4c8b1-openshift-default-58f5449ff-sbp8t"] Nov 03 01:17:24.829625 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:24.829580 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" path="/var/lib/kubelet/pods/dff01bc5-8f42-4d85-af36-90fb2728efe4/volumes" Nov 03 01:17:25.786411 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.786387 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:17:25.792565 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.792537 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-socket\") pod \"c060e707-47c0-4e79-b386-80d66b4f33dc\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " Nov 03 01:17:25.793807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.792862 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-socket" pod "c060e707-47c0-4e79-b386-80d66b4f33dc" (UID: "c060e707-47c0-4e79-b386-80d66b4f33dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:25.793807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.792867 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-kube-api-access-5vsz6\" (UniqueName: \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-kube-api-access-5vsz6\") pod \"c060e707-47c0-4e79-b386-80d66b4f33dc\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " Nov 03 01:17:25.793807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.792967 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-envoy\") pod \"c060e707-47c0-4e79-b386-80d66b4f33dc\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " Nov 03 01:17:25.793807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.793044 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-podinfo\") pod \"c060e707-47c0-4e79-b386-80d66b4f33dc\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " Nov 03 01:17:25.793807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.793076 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-data\") pod \"c060e707-47c0-4e79-b386-80d66b4f33dc\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " Nov 03 01:17:25.793807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.793130 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c060e707-47c0-4e79-b386-80d66b4f33dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c060e707-47c0-4e79-b386-80d66b4f33dc-istiod-ca-cert\") pod \"c060e707-47c0-4e79-b386-80d66b4f33dc\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " Nov 03 01:17:25.793807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.793175 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-token\" (UniqueName: \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-token\") pod \"c060e707-47c0-4e79-b386-80d66b4f33dc\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " Nov 03 01:17:25.793807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.793238 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-credential-socket\") pod \"c060e707-47c0-4e79-b386-80d66b4f33dc\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " Nov 03 01:17:25.793807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.793267 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-certs\") pod \"c060e707-47c0-4e79-b386-80d66b4f33dc\" (UID: \"c060e707-47c0-4e79-b386-80d66b4f33dc\") " Nov 03 01:17:25.793807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.793740 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.794591 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.794561 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-certs" pod "c060e707-47c0-4e79-b386-80d66b4f33dc" (UID: "c060e707-47c0-4e79-b386-80d66b4f33dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:25.795591 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.794749 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-data" pod "c060e707-47c0-4e79-b386-80d66b4f33dc" (UID: "c060e707-47c0-4e79-b386-80d66b4f33dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:25.796303 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.795985 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c060e707-47c0-4e79-b386-80d66b4f33dc-istiod-ca-cert" pod "c060e707-47c0-4e79-b386-80d66b4f33dc" (UID: "c060e707-47c0-4e79-b386-80d66b4f33dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:25.796979 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.796958 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-credential-socket" pod "c060e707-47c0-4e79-b386-80d66b4f33dc" (UID: "c060e707-47c0-4e79-b386-80d66b4f33dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:25.807311 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.807274 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-kube-api-access-5vsz6" pod "c060e707-47c0-4e79-b386-80d66b4f33dc" (UID: "c060e707-47c0-4e79-b386-80d66b4f33dc"). InnerVolumeSpecName "kube-api-access-5vsz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:25.807680 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.807645 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-envoy" pod "c060e707-47c0-4e79-b386-80d66b4f33dc" (UID: "c060e707-47c0-4e79-b386-80d66b4f33dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:25.808210 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.808185 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-podinfo" pod "c060e707-47c0-4e79-b386-80d66b4f33dc" (UID: "c060e707-47c0-4e79-b386-80d66b4f33dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:17:25.811986 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.811960 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-token" pod "c060e707-47c0-4e79-b386-80d66b4f33dc" (UID: "c060e707-47c0-4e79-b386-80d66b4f33dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:25.895295 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.895271 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.895295 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.895293 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.895625 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.895302 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.895625 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.895313 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c060e707-47c0-4e79-b386-80d66b4f33dc-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.895625 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.895321 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.895625 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.895329 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.895625 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.895338 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c060e707-47c0-4e79-b386-80d66b4f33dc-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.895625 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:25.895347 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vsz6\" (UniqueName: \"kubernetes.io/projected/c060e707-47c0-4e79-b386-80d66b4f33dc-kube-api-access-5vsz6\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:26.403226 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:26.403193 4304 generic.go:356] "Generic (PLEG): container finished" podID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerID="d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69" exitCode=0 Nov 03 01:17:26.403401 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:26.403236 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" Nov 03 01:17:26.403401 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:26.403279 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" event={"ID":"c060e707-47c0-4e79-b386-80d66b4f33dc","Type":"ContainerDied","Data":"d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69"} Nov 03 01:17:26.403401 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:26.403333 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5" event={"ID":"c060e707-47c0-4e79-b386-80d66b4f33dc","Type":"ContainerDied","Data":"c6d11756336f1f908312b04cf75d0074b3e08f98ce0afccae9e96a87edf57c4d"} Nov 03 01:17:26.403401 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:26.403360 4304 scope.go:122] "RemoveContainer" containerID="d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69" Nov 03 01:17:26.413423 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:26.413395 4304 scope.go:122] "RemoveContainer" containerID="d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69" Nov 03 01:17:26.413703 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:17:26.413678 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69\": container with ID starting with d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69 not found: ID does not exist" containerID="d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69" Nov 03 01:17:26.413797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:26.413711 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69"} err="failed to get container status \"d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69\": rpc error: code = NotFound desc = could not find container \"d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69\": container with ID starting with d4a940ae4da05ff76cbd806f77bec563e7cdd5ca1a333fd1cc169821c60ffe69 not found: ID does not exist" Nov 03 01:17:26.565953 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:26.565923 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5"] Nov 03 01:17:26.614804 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:26.614764 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jbxs5"] Nov 03 01:17:26.829759 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:26.829679 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" path="/var/lib/kubelet/pods/c060e707-47c0-4e79-b386-80d66b4f33dc/volumes" Nov 03 01:17:28.654279 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:17:28.654246 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:17:48.143590 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.143559 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54"] Nov 03 01:17:48.144060 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144028 4304 cpu_manager.go:414] "containerMap: removing container" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerName="cleanup" Nov 03 01:17:48.144060 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144040 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33faec4b-6d26-45f9-b8bb-104d131d8040" containerName="cleanup" Nov 03 01:17:48.144060 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144056 4304 cpu_manager.go:414] "containerMap: removing container" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerName="istio-proxy" Nov 03 01:17:48.144162 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144062 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerName="istio-proxy" Nov 03 01:17:48.144162 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144077 4304 cpu_manager.go:414] "containerMap: removing container" podUID="4a4c84b0-2ca5-4173-a51a-83116c37bc8e" containerName="maas-api" Nov 03 01:17:48.144162 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144082 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4c84b0-2ca5-4173-a51a-83116c37bc8e" containerName="maas-api" Nov 03 01:17:48.144162 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144100 4304 cpu_manager.go:414] "containerMap: removing container" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" Nov 03 01:17:48.144162 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144104 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" Nov 03 01:17:48.144162 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144110 4304 cpu_manager.go:414] "containerMap: removing container" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" Nov 03 01:17:48.144162 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144115 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" Nov 03 01:17:48.144162 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144122 4304 cpu_manager.go:414] "containerMap: removing container" podUID="7c9c0826-a79b-44de-ab30-37a80157c40d" containerName="payload-processing" Nov 03 01:17:48.144162 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144126 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9c0826-a79b-44de-ab30-37a80157c40d" containerName="payload-processing" Nov 03 01:17:48.144421 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144201 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4c84b0-2ca5-4173-a51a-83116c37bc8e" containerName="maas-api" Nov 03 01:17:48.144421 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144210 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" Nov 03 01:17:48.144421 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144216 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff01bc5-8f42-4d85-af36-90fb2728efe4" containerName="istio-proxy" Nov 03 01:17:48.144421 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144222 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9c0826-a79b-44de-ab30-37a80157c40d" containerName="payload-processing" Nov 03 01:17:48.144421 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.144230 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c060e707-47c0-4e79-b386-80d66b4f33dc" containerName="istio-proxy" Nov 03 01:17:48.147655 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.147640 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.199299 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.199277 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-b2e795-openshift-default-dockercfg-f8cc5\"" Nov 03 01:17:48.218220 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.218197 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080b642d-4c40-441d-9115-3c923bd13683-istio-podinfo\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.218368 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.218230 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080b642d-4c40-441d-9115-3c923bd13683-istiod-ca-cert\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.218368 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.218265 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-certs\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.218477 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.218392 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-istio-token\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.218477 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.218441 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-credential-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.218583 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.218495 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.218583 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.218560 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-data\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.218687 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.218583 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btsdp\" (UniqueName: \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-kube-api-access-btsdp\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.218687 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.218672 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-envoy\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.252564 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.252536 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54"] Nov 03 01:17:48.319921 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.319893 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-data\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320069 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.319923 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btsdp\" (UniqueName: \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-kube-api-access-btsdp\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320069 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.319954 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-envoy\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320069 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.320014 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080b642d-4c40-441d-9115-3c923bd13683-istio-podinfo\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320069 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.320047 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080b642d-4c40-441d-9115-3c923bd13683-istiod-ca-cert\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320248 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.320100 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-certs\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320248 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.320126 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-istio-token\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320248 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.320167 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-credential-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320248 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.320236 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320459 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.320336 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-data\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320512 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.320454 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-certs\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320579 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.320560 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.320618 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.320573 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-credential-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.321070 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.321045 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080b642d-4c40-441d-9115-3c923bd13683-istiod-ca-cert\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.322455 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.322434 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-envoy\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.322732 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.322713 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080b642d-4c40-441d-9115-3c923bd13683-istio-podinfo\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.376507 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.376479 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btsdp\" (UniqueName: \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-kube-api-access-btsdp\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.376657 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.376638 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-istio-token\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.462588 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.462538 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:48.891313 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.891284 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54"] Nov 03 01:17:48.893065 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:17:48.893041 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080b642d_4c40_441d_9115_3c923bd13683.slice/crio-623066d0d74ad0b302d5ee8668791bafcc2f9173314b2c9a54a3e84317f97906 WatchSource:0}: Error finding container 623066d0d74ad0b302d5ee8668791bafcc2f9173314b2c9a54a3e84317f97906: Status 404 returned error can't find the container with id 623066d0d74ad0b302d5ee8668791bafcc2f9173314b2c9a54a3e84317f97906 Nov 03 01:17:48.895684 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.895658 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:17:48.895751 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.895719 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:17:48.895751 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:48.895747 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:17:49.556327 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:49.556284 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" event={"ID":"080b642d-4c40-441d-9115-3c923bd13683","Type":"ContainerStarted","Data":"64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08"} Nov 03 01:17:49.556848 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:49.556348 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" event={"ID":"080b642d-4c40-441d-9115-3c923bd13683","Type":"ContainerStarted","Data":"623066d0d74ad0b302d5ee8668791bafcc2f9173314b2c9a54a3e84317f97906"} Nov 03 01:17:49.771495 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:49.771440 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" podStartSLOduration=1.771424514 podStartE2EDuration="1.771424514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:49.719048424 +0000 UTC m=+4353.002606098" watchObservedRunningTime="2026-11-03 01:17:49.771424514 +0000 UTC m=+4353.054982189" Nov 03 01:17:50.463447 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:50.463412 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:50.464693 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:50.464663 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:17:50.464840 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:50.464708 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" podUID="080b642d-4c40-441d-9115-3c923bd13683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 01:17:51.463829 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:51.463766 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:17:51.464232 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:51.463844 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" podUID="080b642d-4c40-441d-9115-3c923bd13683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 01:17:52.463316 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:52.463283 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:17:52.463497 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:52.463336 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" podUID="080b642d-4c40-441d-9115-3c923bd13683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 01:17:53.463933 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:53.463891 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:17:53.464412 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:53.463954 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" podUID="080b642d-4c40-441d-9115-3c923bd13683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 01:17:54.463237 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:54.463205 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:17:54.463416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:54.463256 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" podUID="080b642d-4c40-441d-9115-3c923bd13683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 01:17:55.467933 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:55.467905 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:55.468373 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:55.468190 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:55.469131 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:17:55.469109 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:17:58.655858 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:17:58.655823 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:18:07.229548 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:07.229520 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2"] Nov 03 01:18:07.229971 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:07.229715 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" containerID="cri-o://b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc" gracePeriod=30 Nov 03 01:18:10.511028 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:10.510996 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:18:10.511404 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:10.511058 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:18:12.500114 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.500054 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:18:12.593797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.593750 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-kube-api-access-cnl9x\" (UniqueName: \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-kube-api-access-cnl9x\") pod \"262aac2d-8690-4618-89f9-32c9e2eb8226\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " Nov 03 01:18:12.593930 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.593843 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262aac2d-8690-4618-89f9-32c9e2eb8226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262aac2d-8690-4618-89f9-32c9e2eb8226-istiod-ca-cert\") pod \"262aac2d-8690-4618-89f9-32c9e2eb8226\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " Nov 03 01:18:12.593930 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.593881 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-socket\") pod \"262aac2d-8690-4618-89f9-32c9e2eb8226\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " Nov 03 01:18:12.594077 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.593931 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-credential-socket\") pod \"262aac2d-8690-4618-89f9-32c9e2eb8226\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " Nov 03 01:18:12.594077 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.593972 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-certs\") pod \"262aac2d-8690-4618-89f9-32c9e2eb8226\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " Nov 03 01:18:12.594077 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594008 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-podinfo\") pod \"262aac2d-8690-4618-89f9-32c9e2eb8226\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " Nov 03 01:18:12.594077 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594046 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-token\" (UniqueName: \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-token\") pod \"262aac2d-8690-4618-89f9-32c9e2eb8226\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " Nov 03 01:18:12.594298 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594099 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-envoy\") pod \"262aac2d-8690-4618-89f9-32c9e2eb8226\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " Nov 03 01:18:12.594298 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594140 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-data\") pod \"262aac2d-8690-4618-89f9-32c9e2eb8226\" (UID: \"262aac2d-8690-4618-89f9-32c9e2eb8226\") " Nov 03 01:18:12.594298 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594260 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-credential-socket" pod "262aac2d-8690-4618-89f9-32c9e2eb8226" (UID: "262aac2d-8690-4618-89f9-32c9e2eb8226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.594469 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594301 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-socket" pod "262aac2d-8690-4618-89f9-32c9e2eb8226" (UID: "262aac2d-8690-4618-89f9-32c9e2eb8226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.594469 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594316 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-certs" pod "262aac2d-8690-4618-89f9-32c9e2eb8226" (UID: "262aac2d-8690-4618-89f9-32c9e2eb8226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.594469 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594367 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262aac2d-8690-4618-89f9-32c9e2eb8226-istiod-ca-cert" pod "262aac2d-8690-4618-89f9-32c9e2eb8226" (UID: "262aac2d-8690-4618-89f9-32c9e2eb8226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:12.594721 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594694 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-data" pod "262aac2d-8690-4618-89f9-32c9e2eb8226" (UID: "262aac2d-8690-4618-89f9-32c9e2eb8226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.594913 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594867 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262aac2d-8690-4618-89f9-32c9e2eb8226-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.594913 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594895 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.594913 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594909 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.595107 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.594921 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.596103 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.596083 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-kube-api-access-cnl9x" pod "262aac2d-8690-4618-89f9-32c9e2eb8226" (UID: "262aac2d-8690-4618-89f9-32c9e2eb8226"). InnerVolumeSpecName "kube-api-access-cnl9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:12.596267 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.596248 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-podinfo" pod "262aac2d-8690-4618-89f9-32c9e2eb8226" (UID: "262aac2d-8690-4618-89f9-32c9e2eb8226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:18:12.596933 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.596913 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-token" pod "262aac2d-8690-4618-89f9-32c9e2eb8226" (UID: "262aac2d-8690-4618-89f9-32c9e2eb8226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:12.596933 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.596914 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-envoy" pod "262aac2d-8690-4618-89f9-32c9e2eb8226" (UID: "262aac2d-8690-4618-89f9-32c9e2eb8226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.696254 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.696221 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnl9x\" (UniqueName: \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-kube-api-access-cnl9x\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.696254 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.696255 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.696470 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.696265 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.696470 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.696273 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.696470 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.696281 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262aac2d-8690-4618-89f9-32c9e2eb8226-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.720727 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.720690 4304 generic.go:356] "Generic (PLEG): container finished" podID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerID="b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc" exitCode=0 Nov 03 01:18:12.720871 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.720729 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" event={"ID":"262aac2d-8690-4618-89f9-32c9e2eb8226","Type":"ContainerDied","Data":"b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc"} Nov 03 01:18:12.720871 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.720742 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" Nov 03 01:18:12.720871 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.720843 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2" event={"ID":"262aac2d-8690-4618-89f9-32c9e2eb8226","Type":"ContainerDied","Data":"f9473afa167738f95f45d906864e5c3a3f0e96192835dbe6fd92597a95b79a42"} Nov 03 01:18:12.720871 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.720865 4304 scope.go:122] "RemoveContainer" containerID="b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc" Nov 03 01:18:12.731684 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.731668 4304 scope.go:122] "RemoveContainer" containerID="b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc" Nov 03 01:18:12.731978 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:18:12.731958 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc\": container with ID starting with b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc not found: ID does not exist" containerID="b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc" Nov 03 01:18:12.732031 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.731986 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc"} err="failed to get container status \"b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc\": rpc error: code = NotFound desc = could not find container \"b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc\": container with ID starting with b5e9bdbdf060eab6c18613dd8f07ebc93f4a0b7171056b46d1c17a7b18768dbc not found: ID does not exist" Nov 03 01:18:12.890908 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.890814 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2"] Nov 03 01:18:12.896231 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:12.896203 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-tsrv2"] Nov 03 01:18:14.828520 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:14.828490 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" path="/var/lib/kubelet/pods/262aac2d-8690-4618-89f9-32c9e2eb8226/volumes" Nov 03 01:18:18.070648 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:18.070616 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x"] Nov 03 01:18:18.182126 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:18.182091 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh"] Nov 03 01:18:23.071550 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:23.071500 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" podUID="10cba01f-ff75-478e-b1db-09d49dd314da" containerName="payload-processing" containerID="cri-o://d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911" gracePeriod=25 Nov 03 01:18:23.183299 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:23.183238 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" podUID="0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1" containerName="payload-pre-processing" containerID="cri-o://40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50" gracePeriod=25 Nov 03 01:18:28.655676 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:18:28.655637 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:18:32.129840 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:32.129801 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" podUID="10cba01f-ff75-478e-b1db-09d49dd314da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:18:32.130242 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:32.129810 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" podUID="0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:18:37.030379 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:37.030338 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj"] Nov 03 01:18:39.726880 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:39.726847 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp"] Nov 03 01:18:39.727302 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:39.727067 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" podUID="321e724f-037e-4159-ab48-4b852a01f18f" containerName="maas-api" containerID="cri-o://b003938afe9cef5cfa8fbbe80f1259dc43c9d3379af3c869681036101ccd6716" gracePeriod=30 Nov 03 01:18:39.882518 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:39.882448 4304 generic.go:356] "Generic (PLEG): container finished" podID="321e724f-037e-4159-ab48-4b852a01f18f" containerID="b003938afe9cef5cfa8fbbe80f1259dc43c9d3379af3c869681036101ccd6716" exitCode=0 Nov 03 01:18:39.882518 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:39.882506 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" event={"ID":"321e724f-037e-4159-ab48-4b852a01f18f","Type":"ContainerDied","Data":"b003938afe9cef5cfa8fbbe80f1259dc43c9d3379af3c869681036101ccd6716"} Nov 03 01:18:39.997946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:39.997923 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 01:18:40.093877 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.093854 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321e724f-037e-4159-ab48-4b852a01f18f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321e724f-037e-4159-ab48-4b852a01f18f-openshift-service-ca\") pod \"321e724f-037e-4159-ab48-4b852a01f18f\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " Nov 03 01:18:40.094009 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.093899 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-maas-api-tls\") pod \"321e724f-037e-4159-ab48-4b852a01f18f\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " Nov 03 01:18:40.094009 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.093931 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321e724f-037e-4159-ab48-4b852a01f18f-kube-api-access-zwmrc\" (UniqueName: \"kubernetes.io/projected/321e724f-037e-4159-ab48-4b852a01f18f-kube-api-access-zwmrc\") pod \"321e724f-037e-4159-ab48-4b852a01f18f\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " Nov 03 01:18:40.094009 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.093985 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-metrics-certs\") pod \"321e724f-037e-4159-ab48-4b852a01f18f\" (UID: \"321e724f-037e-4159-ab48-4b852a01f18f\") " Nov 03 01:18:40.094254 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.094230 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321e724f-037e-4159-ab48-4b852a01f18f-openshift-service-ca" pod "321e724f-037e-4159-ab48-4b852a01f18f" (UID: "321e724f-037e-4159-ab48-4b852a01f18f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:40.094346 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.094327 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321e724f-037e-4159-ab48-4b852a01f18f-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:40.096163 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.096138 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-maas-api-tls" pod "321e724f-037e-4159-ab48-4b852a01f18f" (UID: "321e724f-037e-4159-ab48-4b852a01f18f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:40.096256 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.096156 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-metrics-certs" pod "321e724f-037e-4159-ab48-4b852a01f18f" (UID: "321e724f-037e-4159-ab48-4b852a01f18f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:40.096256 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.096209 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321e724f-037e-4159-ab48-4b852a01f18f-kube-api-access-zwmrc" pod "321e724f-037e-4159-ab48-4b852a01f18f" (UID: "321e724f-037e-4159-ab48-4b852a01f18f"). InnerVolumeSpecName "kube-api-access-zwmrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:40.195434 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.195397 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:40.195434 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.195427 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwmrc\" (UniqueName: \"kubernetes.io/projected/321e724f-037e-4159-ab48-4b852a01f18f-kube-api-access-zwmrc\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:40.195434 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.195439 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321e724f-037e-4159-ab48-4b852a01f18f-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:40.889675 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.889650 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" Nov 03 01:18:40.890039 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.889684 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp" event={"ID":"321e724f-037e-4159-ab48-4b852a01f18f","Type":"ContainerDied","Data":"e722d3addc911ff72e890c6909f63299b28771e46f557b7a83878876e42bbcc5"} Nov 03 01:18:40.890039 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:40.889740 4304 scope.go:122] "RemoveContainer" containerID="b003938afe9cef5cfa8fbbe80f1259dc43c9d3379af3c869681036101ccd6716" Nov 03 01:18:41.050957 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:41.050932 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp"] Nov 03 01:18:41.100195 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:41.100131 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c8ce60-5d646c94d9-bw4kp"] Nov 03 01:18:42.129979 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:42.129946 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" podUID="0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:18:42.129979 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:42.129976 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" podUID="10cba01f-ff75-478e-b1db-09d49dd314da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:18:42.830245 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:42.830211 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321e724f-037e-4159-ab48-4b852a01f18f" path="/var/lib/kubelet/pods/321e724f-037e-4159-ab48-4b852a01f18f/volumes" Nov 03 01:18:43.337826 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.337749 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 01:18:43.427401 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.427371 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10cba01f-ff75-478e-b1db-09d49dd314da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10cba01f-ff75-478e-b1db-09d49dd314da-plugins-config-volume\") pod \"10cba01f-ff75-478e-b1db-09d49dd314da\" (UID: \"10cba01f-ff75-478e-b1db-09d49dd314da\") " Nov 03 01:18:43.427535 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.427411 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cba01f-ff75-478e-b1db-09d49dd314da-kube-api-access-2vqh9\" (UniqueName: \"kubernetes.io/projected/10cba01f-ff75-478e-b1db-09d49dd314da-kube-api-access-2vqh9\") pod \"10cba01f-ff75-478e-b1db-09d49dd314da\" (UID: \"10cba01f-ff75-478e-b1db-09d49dd314da\") " Nov 03 01:18:43.427713 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.427690 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cba01f-ff75-478e-b1db-09d49dd314da-plugins-config-volume" pod "10cba01f-ff75-478e-b1db-09d49dd314da" (UID: "10cba01f-ff75-478e-b1db-09d49dd314da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:43.429660 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.429637 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cba01f-ff75-478e-b1db-09d49dd314da-kube-api-access-2vqh9" pod "10cba01f-ff75-478e-b1db-09d49dd314da" (UID: "10cba01f-ff75-478e-b1db-09d49dd314da"). InnerVolumeSpecName "kube-api-access-2vqh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:43.432333 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.432316 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 01:18:43.528259 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.528209 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-kube-api-access-lfn9n\" (UniqueName: \"kubernetes.io/projected/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-kube-api-access-lfn9n\") pod \"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1\" (UID: \"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1\") " Nov 03 01:18:43.528624 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.528603 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-plugins-config-volume\") pod \"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1\" (UID: \"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1\") " Nov 03 01:18:43.528985 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.528956 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10cba01f-ff75-478e-b1db-09d49dd314da-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:43.528985 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.528964 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-plugins-config-volume" pod "0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1" (UID: "0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:43.528985 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.528977 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vqh9\" (UniqueName: \"kubernetes.io/projected/10cba01f-ff75-478e-b1db-09d49dd314da-kube-api-access-2vqh9\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:43.530521 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.530502 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-kube-api-access-lfn9n" pod "0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1" (UID: "0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1"). InnerVolumeSpecName "kube-api-access-lfn9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:43.629676 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.629613 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfn9n\" (UniqueName: \"kubernetes.io/projected/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-kube-api-access-lfn9n\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:43.629676 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.629637 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:43.909039 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.909005 4304 generic.go:356] "Generic (PLEG): container finished" podID="0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1" containerID="40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50" exitCode=0 Nov 03 01:18:43.909199 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.909062 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" Nov 03 01:18:43.909199 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.909089 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" event={"ID":"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1","Type":"ContainerDied","Data":"40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50"} Nov 03 01:18:43.909199 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.909143 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh" event={"ID":"0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1","Type":"ContainerDied","Data":"9c3c127cf2a91aab21ea7cdd5a285cc5d9adb1a2b299ae9c3edbddcffb8c9a82"} Nov 03 01:18:43.909199 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.909159 4304 scope.go:122] "RemoveContainer" containerID="40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50" Nov 03 01:18:43.911687 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.911661 4304 generic.go:356] "Generic (PLEG): container finished" podID="10cba01f-ff75-478e-b1db-09d49dd314da" containerID="d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911" exitCode=0 Nov 03 01:18:43.911758 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.911737 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" Nov 03 01:18:43.911829 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.911758 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" event={"ID":"10cba01f-ff75-478e-b1db-09d49dd314da","Type":"ContainerDied","Data":"d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911"} Nov 03 01:18:43.911893 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.911830 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x" event={"ID":"10cba01f-ff75-478e-b1db-09d49dd314da","Type":"ContainerDied","Data":"2354606eb4165aaddac00fb44729b79dac22d437f018c333310997e4fa8041ec"} Nov 03 01:18:43.920923 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.920898 4304 scope.go:122] "RemoveContainer" containerID="40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50" Nov 03 01:18:43.921346 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:18:43.921327 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50\": container with ID starting with 40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50 not found: ID does not exist" containerID="40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50" Nov 03 01:18:43.921458 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.921358 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50"} err="failed to get container status \"40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50\": rpc error: code = NotFound desc = could not find container \"40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50\": container with ID starting with 40bf05bd76740ddf20af600319da5ca96bbd4cf6983b535479033a1e4890ab50 not found: ID does not exist" Nov 03 01:18:43.921458 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.921381 4304 scope.go:122] "RemoveContainer" containerID="d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911" Nov 03 01:18:43.930227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.930212 4304 scope.go:122] "RemoveContainer" containerID="d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911" Nov 03 01:18:43.930499 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:18:43.930459 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911\": container with ID starting with d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911 not found: ID does not exist" containerID="d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911" Nov 03 01:18:43.930499 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:43.930483 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911"} err="failed to get container status \"d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911\": rpc error: code = NotFound desc = could not find container \"d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911\": container with ID starting with d0bb8fdbfc358c383b82bcde37cc380b2994f9776eafa69f80c6cd66b0a34911 not found: ID does not exist" Nov 03 01:18:44.071172 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:44.071146 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x"] Nov 03 01:18:44.121358 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:44.121335 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c8ce60-bff96678c-fjf5x"] Nov 03 01:18:44.181328 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:44.181268 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh"] Nov 03 01:18:44.185027 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:44.185007 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c8ce60-76895945dc-swgkh"] Nov 03 01:18:44.829823 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:44.829764 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1" path="/var/lib/kubelet/pods/0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1/volumes" Nov 03 01:18:44.830368 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:44.830322 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10cba01f-ff75-478e-b1db-09d49dd314da" path="/var/lib/kubelet/pods/10cba01f-ff75-478e-b1db-09d49dd314da/volumes" Nov 03 01:18:48.454708 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:48.454674 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn"] Nov 03 01:18:48.455155 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:48.454937 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" containerID="cri-o://a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034" gracePeriod=30 Nov 03 01:18:50.636218 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:50.636171 4304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:18:50.636617 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:50.636259 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:18:52.053541 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.053489 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" podUID="082c2451-3bb7-4b3a-82c0-f8ea634adcdb" containerName="main" containerID="cri-o://7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766" gracePeriod=45 Nov 03 01:18:52.187807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.187752 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" podUID="082c2451-3bb7-4b3a-82c0-f8ea634adcdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused" Nov 03 01:18:52.314462 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.314407 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 01:18:52.339659 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.339631 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tls-certs\" (UniqueName: \"kubernetes.io/secret/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tls-certs\") pod \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " Nov 03 01:18:52.339824 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.339689 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-kube-api-access-kfk7w\" (UniqueName: \"kubernetes.io/projected/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-kube-api-access-kfk7w\") pod \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " Nov 03 01:18:52.339824 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.339726 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-home\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-home\") pod \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " Nov 03 01:18:52.339824 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.339764 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-dshm\") pod \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " Nov 03 01:18:52.340007 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.339856 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tmp-dir\") pod \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " Nov 03 01:18:52.340007 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.339889 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-model-cache\") pod \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\" (UID: \"082c2451-3bb7-4b3a-82c0-f8ea634adcdb\") " Nov 03 01:18:52.340108 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.340000 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-home" pod "082c2451-3bb7-4b3a-82c0-f8ea634adcdb" (UID: "082c2451-3bb7-4b3a-82c0-f8ea634adcdb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.340179 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.340131 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tmp-dir" pod "082c2451-3bb7-4b3a-82c0-f8ea634adcdb" (UID: "082c2451-3bb7-4b3a-82c0-f8ea634adcdb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.340229 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.340199 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-model-cache" pod "082c2451-3bb7-4b3a-82c0-f8ea634adcdb" (UID: "082c2451-3bb7-4b3a-82c0-f8ea634adcdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.340419 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.340349 4304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-home\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.340419 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.340368 4304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tmp-dir\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.340419 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.340376 4304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-model-cache\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.342047 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.342024 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tls-certs" pod "082c2451-3bb7-4b3a-82c0-f8ea634adcdb" (UID: "082c2451-3bb7-4b3a-82c0-f8ea634adcdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:52.342136 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.342027 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-dshm" pod "082c2451-3bb7-4b3a-82c0-f8ea634adcdb" (UID: "082c2451-3bb7-4b3a-82c0-f8ea634adcdb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.342136 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.342066 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-kube-api-access-kfk7w" pod "082c2451-3bb7-4b3a-82c0-f8ea634adcdb" (UID: "082c2451-3bb7-4b3a-82c0-f8ea634adcdb"). InnerVolumeSpecName "kube-api-access-kfk7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:52.444520 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.444481 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfk7w\" (UniqueName: \"kubernetes.io/projected/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-kube-api-access-kfk7w\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.444520 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.444518 4304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-dshm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.444695 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:52.444532 4304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082c2451-3bb7-4b3a-82c0-f8ea634adcdb-tls-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:53.019099 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.019069 4304 generic.go:356] "Generic (PLEG): container finished" podID="082c2451-3bb7-4b3a-82c0-f8ea634adcdb" containerID="7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766" exitCode=0 Nov 03 01:18:53.019270 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.019118 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" event={"ID":"082c2451-3bb7-4b3a-82c0-f8ea634adcdb","Type":"ContainerDied","Data":"7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766"} Nov 03 01:18:53.019270 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.019125 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" Nov 03 01:18:53.019270 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.019146 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj" event={"ID":"082c2451-3bb7-4b3a-82c0-f8ea634adcdb","Type":"ContainerDied","Data":"9de8f95230a1d8c6b9bab0d0a0e2dd5b3c99ac30adf5aa4c801b9b9284ed70fd"} Nov 03 01:18:53.019270 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.019162 4304 scope.go:122] "RemoveContainer" containerID="7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766" Nov 03 01:18:53.028423 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.028406 4304 scope.go:122] "RemoveContainer" containerID="7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766" Nov 03 01:18:53.028681 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:18:53.028663 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766\": container with ID starting with 7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766 not found: ID does not exist" containerID="7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766" Nov 03 01:18:53.028742 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.028687 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766"} err="failed to get container status \"7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766\": rpc error: code = NotFound desc = could not find container \"7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766\": container with ID starting with 7e8d0e9ec9292faf63d768024653232a335a6fe9e1ea67c354c99da52f69a766 not found: ID does not exist" Nov 03 01:18:53.181120 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.181094 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj"] Nov 03 01:18:53.184236 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.184214 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/simulator-w3-d7bfc2-kserve-6c46fb4c54-dpqsj"] Nov 03 01:18:53.722439 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.722383 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 01:18:53.756438 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756413 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1c30b9-b6df-411c-8290-ff0a313629f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1c30b9-b6df-411c-8290-ff0a313629f7-istiod-ca-cert\") pod \"4c1c30b9-b6df-411c-8290-ff0a313629f7\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " Nov 03 01:18:53.756569 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756461 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-kube-api-access-8xzfs\" (UniqueName: \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-kube-api-access-8xzfs\") pod \"4c1c30b9-b6df-411c-8290-ff0a313629f7\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " Nov 03 01:18:53.756569 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756482 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-certs\") pod \"4c1c30b9-b6df-411c-8290-ff0a313629f7\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " Nov 03 01:18:53.756569 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756532 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-envoy\") pod \"4c1c30b9-b6df-411c-8290-ff0a313629f7\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " Nov 03 01:18:53.756756 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756596 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-socket\") pod \"4c1c30b9-b6df-411c-8290-ff0a313629f7\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " Nov 03 01:18:53.756756 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756633 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-data\") pod \"4c1c30b9-b6df-411c-8290-ff0a313629f7\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " Nov 03 01:18:53.756756 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756723 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-credential-socket\") pod \"4c1c30b9-b6df-411c-8290-ff0a313629f7\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " Nov 03 01:18:53.756756 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756733 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1c30b9-b6df-411c-8290-ff0a313629f7-istiod-ca-cert" pod "4c1c30b9-b6df-411c-8290-ff0a313629f7" (UID: "4c1c30b9-b6df-411c-8290-ff0a313629f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:53.757024 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756762 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-token\" (UniqueName: \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-token\") pod \"4c1c30b9-b6df-411c-8290-ff0a313629f7\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " Nov 03 01:18:53.757024 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756850 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-podinfo\") pod \"4c1c30b9-b6df-411c-8290-ff0a313629f7\" (UID: \"4c1c30b9-b6df-411c-8290-ff0a313629f7\") " Nov 03 01:18:53.757024 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756899 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-certs" pod "4c1c30b9-b6df-411c-8290-ff0a313629f7" (UID: "4c1c30b9-b6df-411c-8290-ff0a313629f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:53.757024 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.756909 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-socket" pod "4c1c30b9-b6df-411c-8290-ff0a313629f7" (UID: "4c1c30b9-b6df-411c-8290-ff0a313629f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:53.757396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.757028 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-credential-socket" pod "4c1c30b9-b6df-411c-8290-ff0a313629f7" (UID: "4c1c30b9-b6df-411c-8290-ff0a313629f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:53.757396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.757226 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-data" pod "4c1c30b9-b6df-411c-8290-ff0a313629f7" (UID: "4c1c30b9-b6df-411c-8290-ff0a313629f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:53.757396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.757296 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:53.757396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.757313 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1c30b9-b6df-411c-8290-ff0a313629f7-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:53.757396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.757325 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:53.757396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.757336 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:53.757396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.757344 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:53.758766 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.758738 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-envoy" pod "4c1c30b9-b6df-411c-8290-ff0a313629f7" (UID: "4c1c30b9-b6df-411c-8290-ff0a313629f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:53.758901 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.758879 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-kube-api-access-8xzfs" pod "4c1c30b9-b6df-411c-8290-ff0a313629f7" (UID: "4c1c30b9-b6df-411c-8290-ff0a313629f7"). InnerVolumeSpecName "kube-api-access-8xzfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:53.759352 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.759330 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-podinfo" pod "4c1c30b9-b6df-411c-8290-ff0a313629f7" (UID: "4c1c30b9-b6df-411c-8290-ff0a313629f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:18:53.759352 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.759346 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-token" pod "4c1c30b9-b6df-411c-8290-ff0a313629f7" (UID: "4c1c30b9-b6df-411c-8290-ff0a313629f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:53.858426 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.858398 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:53.858426 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.858422 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:53.858563 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.858432 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1c30b9-b6df-411c-8290-ff0a313629f7-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:53.858563 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:53.858442 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xzfs\" (UniqueName: \"kubernetes.io/projected/4c1c30b9-b6df-411c-8290-ff0a313629f7-kube-api-access-8xzfs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:54.027876 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.027800 4304 generic.go:356] "Generic (PLEG): container finished" podID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerID="a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034" exitCode=0 Nov 03 01:18:54.027876 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.027849 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" Nov 03 01:18:54.028182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.027876 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" event={"ID":"4c1c30b9-b6df-411c-8290-ff0a313629f7","Type":"ContainerDied","Data":"a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034"} Nov 03 01:18:54.028182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.027932 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn" event={"ID":"4c1c30b9-b6df-411c-8290-ff0a313629f7","Type":"ContainerDied","Data":"3561b29a61cadb55f5acf5cf03d0b2c99f5f69155f783468fdecdc3e69a84914"} Nov 03 01:18:54.028182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.027964 4304 scope.go:122] "RemoveContainer" containerID="a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034" Nov 03 01:18:54.038332 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.038317 4304 scope.go:122] "RemoveContainer" containerID="f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74" Nov 03 01:18:54.046899 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.046880 4304 scope.go:122] "RemoveContainer" containerID="a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034" Nov 03 01:18:54.047144 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:18:54.047125 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034\": container with ID starting with a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034 not found: ID does not exist" containerID="a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034" Nov 03 01:18:54.047186 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.047153 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034"} err="failed to get container status \"a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034\": rpc error: code = NotFound desc = could not find container \"a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034\": container with ID starting with a19161797446c1a82a0a7a5e9d4ae6edc062d82db2a4f7efacfb818c86669034 not found: ID does not exist" Nov 03 01:18:54.047186 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.047170 4304 scope.go:122] "RemoveContainer" containerID="f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74" Nov 03 01:18:54.047396 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:18:54.047379 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74\": container with ID starting with f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74 not found: ID does not exist" containerID="f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74" Nov 03 01:18:54.047434 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.047402 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74"} err="failed to get container status \"f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74\": rpc error: code = NotFound desc = could not find container \"f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74\": container with ID starting with f77db38811d3c36d9618f5df5c6097d3c3123a88b4daa55801b7e46f2cf73f74 not found: ID does not exist" Nov 03 01:18:54.190285 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.190262 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn"] Nov 03 01:18:54.193423 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.193403 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c8ce60-openshift-default-696d795d7c-rvvsn"] Nov 03 01:18:54.829622 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.829590 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="082c2451-3bb7-4b3a-82c0-f8ea634adcdb" path="/var/lib/kubelet/pods/082c2451-3bb7-4b3a-82c0-f8ea634adcdb/volumes" Nov 03 01:18:54.830140 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:18:54.830117 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" path="/var/lib/kubelet/pods/4c1c30b9-b6df-411c-8290-ff0a313629f7/volumes" Nov 03 01:18:58.654372 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:18:58.654335 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:19:28.654170 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:19:28.654133 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:19:30.299039 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:30.298967 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l"] Nov 03 01:19:30.299428 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:30.299183 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" containerID="cri-o://7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b" gracePeriod=30 Nov 03 01:19:34.313416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:34.313381 4304 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:19:34.313806 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:34.313447 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:19:35.559436 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.559414 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 01:19:35.644129 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644107 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-kube-api-access-58twv\" (UniqueName: \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-kube-api-access-58twv\") pod \"bf51c557-8e65-4a4d-8746-35df22b465e6\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " Nov 03 01:19:35.644267 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644141 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-socket\") pod \"bf51c557-8e65-4a4d-8746-35df22b465e6\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " Nov 03 01:19:35.644267 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644171 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-podinfo\") pod \"bf51c557-8e65-4a4d-8746-35df22b465e6\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " Nov 03 01:19:35.644267 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644255 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-certs\") pod \"bf51c557-8e65-4a4d-8746-35df22b465e6\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " Nov 03 01:19:35.644433 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644290 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-credential-socket\") pod \"bf51c557-8e65-4a4d-8746-35df22b465e6\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " Nov 03 01:19:35.644433 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644327 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-token\" (UniqueName: \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-token\") pod \"bf51c557-8e65-4a4d-8746-35df22b465e6\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " Nov 03 01:19:35.644433 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644360 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-envoy\") pod \"bf51c557-8e65-4a4d-8746-35df22b465e6\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " Nov 03 01:19:35.644433 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644405 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf51c557-8e65-4a4d-8746-35df22b465e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf51c557-8e65-4a4d-8746-35df22b465e6-istiod-ca-cert\") pod \"bf51c557-8e65-4a4d-8746-35df22b465e6\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " Nov 03 01:19:35.644654 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644429 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-socket" pod "bf51c557-8e65-4a4d-8746-35df22b465e6" (UID: "bf51c557-8e65-4a4d-8746-35df22b465e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:35.644654 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644441 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-data\") pod \"bf51c557-8e65-4a4d-8746-35df22b465e6\" (UID: \"bf51c557-8e65-4a4d-8746-35df22b465e6\") " Nov 03 01:19:35.644654 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644548 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-certs" pod "bf51c557-8e65-4a4d-8746-35df22b465e6" (UID: "bf51c557-8e65-4a4d-8746-35df22b465e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:35.644654 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644615 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-credential-socket" pod "bf51c557-8e65-4a4d-8746-35df22b465e6" (UID: "bf51c557-8e65-4a4d-8746-35df22b465e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:35.645068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644683 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-data" pod "bf51c557-8e65-4a4d-8746-35df22b465e6" (UID: "bf51c557-8e65-4a4d-8746-35df22b465e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:35.645068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644933 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf51c557-8e65-4a4d-8746-35df22b465e6-istiod-ca-cert" pod "bf51c557-8e65-4a4d-8746-35df22b465e6" (UID: "bf51c557-8e65-4a4d-8746-35df22b465e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:35.645068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.644989 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:35.645068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.645012 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:35.645068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.645027 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:35.645068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.645042 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:35.646397 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.646373 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-kube-api-access-58twv" pod "bf51c557-8e65-4a4d-8746-35df22b465e6" (UID: "bf51c557-8e65-4a4d-8746-35df22b465e6"). InnerVolumeSpecName "kube-api-access-58twv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:35.646517 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.646486 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-podinfo" pod "bf51c557-8e65-4a4d-8746-35df22b465e6" (UID: "bf51c557-8e65-4a4d-8746-35df22b465e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:19:35.646832 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.646808 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-token" pod "bf51c557-8e65-4a4d-8746-35df22b465e6" (UID: "bf51c557-8e65-4a4d-8746-35df22b465e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:35.646946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.646847 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-envoy" pod "bf51c557-8e65-4a4d-8746-35df22b465e6" (UID: "bf51c557-8e65-4a4d-8746-35df22b465e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:35.746143 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.746110 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:35.746143 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.746138 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:35.746143 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.746147 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf51c557-8e65-4a4d-8746-35df22b465e6-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:35.746363 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.746158 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58twv\" (UniqueName: \"kubernetes.io/projected/bf51c557-8e65-4a4d-8746-35df22b465e6-kube-api-access-58twv\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:35.746363 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:35.746169 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf51c557-8e65-4a4d-8746-35df22b465e6-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.271758 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.271715 4304 generic.go:356] "Generic (PLEG): container finished" podID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerID="7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b" exitCode=0 Nov 03 01:19:36.271941 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.271753 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" event={"ID":"bf51c557-8e65-4a4d-8746-35df22b465e6","Type":"ContainerDied","Data":"7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b"} Nov 03 01:19:36.271941 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.271770 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" Nov 03 01:19:36.271941 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.271827 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l" event={"ID":"bf51c557-8e65-4a4d-8746-35df22b465e6","Type":"ContainerDied","Data":"38fa61bab29dfb8a6cd77e886ee3526af1dc822c174783c69217706fe0107182"} Nov 03 01:19:36.271941 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.271853 4304 scope.go:122] "RemoveContainer" containerID="7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b" Nov 03 01:19:36.282024 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.282003 4304 scope.go:122] "RemoveContainer" containerID="3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856" Nov 03 01:19:36.291087 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.291071 4304 scope.go:122] "RemoveContainer" containerID="7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b" Nov 03 01:19:36.291377 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:19:36.291358 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b\": container with ID starting with 7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b not found: ID does not exist" containerID="7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b" Nov 03 01:19:36.291438 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.291385 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b"} err="failed to get container status \"7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b\": rpc error: code = NotFound desc = could not find container \"7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b\": container with ID starting with 7c438a6897b9ac9d012eb8c4b181da4275ebe1a90e975887a4a3fbaeb47a8a2b not found: ID does not exist" Nov 03 01:19:36.291438 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.291402 4304 scope.go:122] "RemoveContainer" containerID="3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856" Nov 03 01:19:36.291685 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:19:36.291666 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856\": container with ID starting with 3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856 not found: ID does not exist" containerID="3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856" Nov 03 01:19:36.291760 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.291696 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856"} err="failed to get container status \"3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856\": rpc error: code = NotFound desc = could not find container \"3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856\": container with ID starting with 3abbfe6871ae225b43809c40754f28f39aab2691e60344a4b565db61b8e90856 not found: ID does not exist" Nov 03 01:19:36.434339 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.434307 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l"] Nov 03 01:19:36.483829 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.483800 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wlm7l"] Nov 03 01:19:36.830970 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:36.830935 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" path="/var/lib/kubelet/pods/bf51c557-8e65-4a4d-8746-35df22b465e6/volumes" Nov 03 01:19:51.805893 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:51.805856 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq"] Nov 03 01:19:54.672375 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:19:54.672346 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85"] Nov 03 01:19:58.655210 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:19:58.655176 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:20:05.204255 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:05.204211 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk"] Nov 03 01:20:06.826848 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:06.826764 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" podUID="26d57814-f91a-423c-ab38-5cb55e4d77a0" containerName="main" containerID="cri-o://1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad" gracePeriod=45 Nov 03 01:20:06.909429 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:06.909398 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" podUID="26d57814-f91a-423c-ab38-5cb55e4d77a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused" Nov 03 01:20:07.081005 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.080946 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 01:20:07.184613 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.184587 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-tmp-dir\") pod \"26d57814-f91a-423c-ab38-5cb55e4d77a0\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " Nov 03 01:20:07.184807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.184698 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26d57814-f91a-423c-ab38-5cb55e4d77a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/26d57814-f91a-423c-ab38-5cb55e4d77a0-tls-certs\") pod \"26d57814-f91a-423c-ab38-5cb55e4d77a0\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " Nov 03 01:20:07.184807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.184735 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-home\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-home\") pod \"26d57814-f91a-423c-ab38-5cb55e4d77a0\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " Nov 03 01:20:07.184968 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.184827 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-model-cache\") pod \"26d57814-f91a-423c-ab38-5cb55e4d77a0\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " Nov 03 01:20:07.184968 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.184872 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-dshm\") pod \"26d57814-f91a-423c-ab38-5cb55e4d77a0\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " Nov 03 01:20:07.184968 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.184870 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-tmp-dir" pod "26d57814-f91a-423c-ab38-5cb55e4d77a0" (UID: "26d57814-f91a-423c-ab38-5cb55e4d77a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.184968 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.184906 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d57814-f91a-423c-ab38-5cb55e4d77a0-kube-api-access-qzxmc\" (UniqueName: \"kubernetes.io/projected/26d57814-f91a-423c-ab38-5cb55e4d77a0-kube-api-access-qzxmc\") pod \"26d57814-f91a-423c-ab38-5cb55e4d77a0\" (UID: \"26d57814-f91a-423c-ab38-5cb55e4d77a0\") " Nov 03 01:20:07.185173 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.185096 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-home" pod "26d57814-f91a-423c-ab38-5cb55e4d77a0" (UID: "26d57814-f91a-423c-ab38-5cb55e4d77a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.185173 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.185108 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-model-cache" pod "26d57814-f91a-423c-ab38-5cb55e4d77a0" (UID: "26d57814-f91a-423c-ab38-5cb55e4d77a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.185354 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.185329 4304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-tmp-dir\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.185354 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.185355 4304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-home\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.185540 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.185367 4304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-model-cache\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.187849 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.187823 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d57814-f91a-423c-ab38-5cb55e4d77a0-kube-api-access-qzxmc" pod "26d57814-f91a-423c-ab38-5cb55e4d77a0" (UID: "26d57814-f91a-423c-ab38-5cb55e4d77a0"). InnerVolumeSpecName "kube-api-access-qzxmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:07.187976 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.187825 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-dshm" pod "26d57814-f91a-423c-ab38-5cb55e4d77a0" (UID: "26d57814-f91a-423c-ab38-5cb55e4d77a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.188170 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.188147 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26d57814-f91a-423c-ab38-5cb55e4d77a0-tls-certs" pod "26d57814-f91a-423c-ab38-5cb55e4d77a0" (UID: "26d57814-f91a-423c-ab38-5cb55e4d77a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:07.286367 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.286336 4304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d57814-f91a-423c-ab38-5cb55e4d77a0-dshm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.286367 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.286366 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzxmc\" (UniqueName: \"kubernetes.io/projected/26d57814-f91a-423c-ab38-5cb55e4d77a0-kube-api-access-qzxmc\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.286564 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.286380 4304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d57814-f91a-423c-ab38-5cb55e4d77a0-tls-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.456979 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.456935 4304 generic.go:356] "Generic (PLEG): container finished" podID="26d57814-f91a-423c-ab38-5cb55e4d77a0" containerID="1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad" exitCode=0 Nov 03 01:20:07.457168 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.456982 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" Nov 03 01:20:07.457168 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.456999 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" event={"ID":"26d57814-f91a-423c-ab38-5cb55e4d77a0","Type":"ContainerDied","Data":"1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad"} Nov 03 01:20:07.457168 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.457035 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq" event={"ID":"26d57814-f91a-423c-ab38-5cb55e4d77a0","Type":"ContainerDied","Data":"520771a47aab8fec22235c67b98940c0cec558152c96d73bf07f1eeb62e49b91"} Nov 03 01:20:07.457168 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.457056 4304 scope.go:122] "RemoveContainer" containerID="1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad" Nov 03 01:20:07.468922 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.468903 4304 scope.go:122] "RemoveContainer" containerID="1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad" Nov 03 01:20:07.469203 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:20:07.469183 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad\": container with ID starting with 1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad not found: ID does not exist" containerID="1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad" Nov 03 01:20:07.469282 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.469211 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad"} err="failed to get container status \"1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad\": rpc error: code = NotFound desc = could not find container \"1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad\": container with ID starting with 1f7b28aa6229ab9553f8d3bcaa68f32579d783ca56455c909f796c29a5d2b5ad not found: ID does not exist" Nov 03 01:20:07.621879 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.621852 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq"] Nov 03 01:20:07.625296 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:07.625263 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/premium-w2-531ee4-kserve-7d445c97fc-rvvfq"] Nov 03 01:20:08.829190 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:08.829160 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d57814-f91a-423c-ab38-5cb55e4d77a0" path="/var/lib/kubelet/pods/26d57814-f91a-423c-ab38-5cb55e4d77a0/volumes" Nov 03 01:20:09.691921 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:09.691868 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" podUID="30838ca2-820a-4d11-8ce5-5dfd2d579bf8" containerName="main" containerID="cri-o://7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74" gracePeriod=45 Nov 03 01:20:09.947151 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:09.947098 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:20:10.014675 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.014652 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-kube-api-access-kf66s\" (UniqueName: \"kubernetes.io/projected/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-kube-api-access-kf66s\") pod \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " Nov 03 01:20:10.014844 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.014820 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-home\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-home\") pod \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " Nov 03 01:20:10.014920 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.014906 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tmp-dir\") pod \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " Nov 03 01:20:10.014958 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.014942 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tls-certs\" (UniqueName: \"kubernetes.io/secret/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tls-certs\") pod \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " Nov 03 01:20:10.015006 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.014990 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-dshm\") pod \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " Nov 03 01:20:10.015063 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.015049 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-model-cache\") pod \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\" (UID: \"30838ca2-820a-4d11-8ce5-5dfd2d579bf8\") " Nov 03 01:20:10.015116 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.015084 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-home" pod "30838ca2-820a-4d11-8ce5-5dfd2d579bf8" (UID: "30838ca2-820a-4d11-8ce5-5dfd2d579bf8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:10.015224 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.015150 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tmp-dir" pod "30838ca2-820a-4d11-8ce5-5dfd2d579bf8" (UID: "30838ca2-820a-4d11-8ce5-5dfd2d579bf8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:10.015402 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.015382 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-model-cache" pod "30838ca2-820a-4d11-8ce5-5dfd2d579bf8" (UID: "30838ca2-820a-4d11-8ce5-5dfd2d579bf8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:10.015693 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.015669 4304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-home\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:10.015751 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.015700 4304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tmp-dir\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:10.015751 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.015714 4304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-model-cache\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:10.017162 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.017142 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-dshm" pod "30838ca2-820a-4d11-8ce5-5dfd2d579bf8" (UID: "30838ca2-820a-4d11-8ce5-5dfd2d579bf8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:10.017227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.017171 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-kube-api-access-kf66s" pod "30838ca2-820a-4d11-8ce5-5dfd2d579bf8" (UID: "30838ca2-820a-4d11-8ce5-5dfd2d579bf8"). InnerVolumeSpecName "kube-api-access-kf66s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:10.017227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.017203 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tls-certs" pod "30838ca2-820a-4d11-8ce5-5dfd2d579bf8" (UID: "30838ca2-820a-4d11-8ce5-5dfd2d579bf8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:10.117174 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.117145 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf66s\" (UniqueName: \"kubernetes.io/projected/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-kube-api-access-kf66s\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:10.117174 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.117172 4304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-tls-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:10.117340 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.117183 4304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30838ca2-820a-4d11-8ce5-5dfd2d579bf8-dshm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:10.475684 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.475652 4304 generic.go:356] "Generic (PLEG): container finished" podID="30838ca2-820a-4d11-8ce5-5dfd2d579bf8" containerID="7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74" exitCode=0 Nov 03 01:20:10.475850 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.475703 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" Nov 03 01:20:10.475850 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.475735 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" event={"ID":"30838ca2-820a-4d11-8ce5-5dfd2d579bf8","Type":"ContainerDied","Data":"7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74"} Nov 03 01:20:10.475850 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.475793 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85" event={"ID":"30838ca2-820a-4d11-8ce5-5dfd2d579bf8","Type":"ContainerDied","Data":"7a398577941d6ea5e88cdc227d5154fa0dce94f68fa8d2154c2f08142d224a0e"} Nov 03 01:20:10.475850 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.475814 4304 scope.go:122] "RemoveContainer" containerID="7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74" Nov 03 01:20:10.486201 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.486185 4304 scope.go:122] "RemoveContainer" containerID="7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74" Nov 03 01:20:10.486472 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:20:10.486454 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74\": container with ID starting with 7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74 not found: ID does not exist" containerID="7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74" Nov 03 01:20:10.486528 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.486480 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74"} err="failed to get container status \"7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74\": rpc error: code = NotFound desc = could not find container \"7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74\": container with ID starting with 7fa19c5a81a21bfd41755b3986c713c163a27d0799819c93bba6ce1db0f95e74 not found: ID does not exist" Nov 03 01:20:10.637696 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.637667 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85"] Nov 03 01:20:10.643478 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.643458 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/rate-test-model-fce94b-kserve-7f65496797-jlw85"] Nov 03 01:20:10.828545 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:10.828480 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30838ca2-820a-4d11-8ce5-5dfd2d579bf8" path="/var/lib/kubelet/pods/30838ca2-820a-4d11-8ce5-5dfd2d579bf8/volumes" Nov 03 01:20:20.224286 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.224236 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" podUID="08ce4326-4703-4061-aec4-17b4c6567785" containerName="main" containerID="cri-o://738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101" gracePeriod=45 Nov 03 01:20:20.366370 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.366334 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" podUID="08ce4326-4703-4061-aec4-17b4c6567785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.161:8000/ready\": dial tcp 10.133.0.161:8000: connect: connection refused" Nov 03 01:20:20.477033 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.476978 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:20:20.546094 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.546062 4304 generic.go:356] "Generic (PLEG): container finished" podID="08ce4326-4703-4061-aec4-17b4c6567785" containerID="738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101" exitCode=0 Nov 03 01:20:20.546304 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.546143 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" Nov 03 01:20:20.546304 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.546158 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" event={"ID":"08ce4326-4703-4061-aec4-17b4c6567785","Type":"ContainerDied","Data":"738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101"} Nov 03 01:20:20.546304 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.546198 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk" event={"ID":"08ce4326-4703-4061-aec4-17b4c6567785","Type":"ContainerDied","Data":"a77ef79051754a80da750f5a310040ef6db64ce4b1cd5cb22838da3d9a7c0eeb"} Nov 03 01:20:20.546304 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.546218 4304 scope.go:122] "RemoveContainer" containerID="738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101" Nov 03 01:20:20.557081 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.557045 4304 scope.go:122] "RemoveContainer" containerID="738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101" Nov 03 01:20:20.557622 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:20:20.557590 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101\": container with ID starting with 738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101 not found: ID does not exist" containerID="738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101" Nov 03 01:20:20.557717 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.557633 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101"} err="failed to get container status \"738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101\": rpc error: code = NotFound desc = could not find container \"738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101\": container with ID starting with 738acb98ab98babea484a2c2e1026c7242b3926881d3c547115237c89479f101 not found: ID does not exist" Nov 03 01:20:20.625879 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.625840 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-tmp-dir\") pod \"08ce4326-4703-4061-aec4-17b4c6567785\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " Nov 03 01:20:20.626049 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.625896 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ce4326-4703-4061-aec4-17b4c6567785-kube-api-access-84pzv\" (UniqueName: \"kubernetes.io/projected/08ce4326-4703-4061-aec4-17b4c6567785-kube-api-access-84pzv\") pod \"08ce4326-4703-4061-aec4-17b4c6567785\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " Nov 03 01:20:20.626049 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.625972 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ce4326-4703-4061-aec4-17b4c6567785-tls-certs\" (UniqueName: \"kubernetes.io/secret/08ce4326-4703-4061-aec4-17b4c6567785-tls-certs\") pod \"08ce4326-4703-4061-aec4-17b4c6567785\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " Nov 03 01:20:20.626182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.626063 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-home\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-home\") pod \"08ce4326-4703-4061-aec4-17b4c6567785\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " Nov 03 01:20:20.626182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.626109 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-dshm\") pod \"08ce4326-4703-4061-aec4-17b4c6567785\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " Nov 03 01:20:20.626182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.626150 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-model-cache\") pod \"08ce4326-4703-4061-aec4-17b4c6567785\" (UID: \"08ce4326-4703-4061-aec4-17b4c6567785\") " Nov 03 01:20:20.626505 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.626460 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-home" pod "08ce4326-4703-4061-aec4-17b4c6567785" (UID: "08ce4326-4703-4061-aec4-17b4c6567785"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:20.626626 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.626600 4304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-home\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:20.626722 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.626692 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-model-cache" pod "08ce4326-4703-4061-aec4-17b4c6567785" (UID: "08ce4326-4703-4061-aec4-17b4c6567785"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:20.627136 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.627113 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-tmp-dir" pod "08ce4326-4703-4061-aec4-17b4c6567785" (UID: "08ce4326-4703-4061-aec4-17b4c6567785"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:20.632992 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.632966 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ce4326-4703-4061-aec4-17b4c6567785-tls-certs" pod "08ce4326-4703-4061-aec4-17b4c6567785" (UID: "08ce4326-4703-4061-aec4-17b4c6567785"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:20.639915 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.639889 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-dshm" pod "08ce4326-4703-4061-aec4-17b4c6567785" (UID: "08ce4326-4703-4061-aec4-17b4c6567785"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:20.640036 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.640015 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ce4326-4703-4061-aec4-17b4c6567785-kube-api-access-84pzv" pod "08ce4326-4703-4061-aec4-17b4c6567785" (UID: "08ce4326-4703-4061-aec4-17b4c6567785"). InnerVolumeSpecName "kube-api-access-84pzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:20.727624 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.727535 4304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-tmp-dir\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:20.727624 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.727571 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84pzv\" (UniqueName: \"kubernetes.io/projected/08ce4326-4703-4061-aec4-17b4c6567785-kube-api-access-84pzv\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:20.727624 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.727589 4304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ce4326-4703-4061-aec4-17b4c6567785-tls-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:20.727946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.727635 4304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-dshm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:20.727946 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.727650 4304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ce4326-4703-4061-aec4-17b4c6567785-model-cache\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:20.962084 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.962051 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk"] Nov 03 01:20:20.965937 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:20.965914 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/rate-test-model-20f0a3-kserve-69f6888cf7-lwgnk"] Nov 03 01:20:22.828225 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:22.828191 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ce4326-4703-4061-aec4-17b4c6567785" path="/var/lib/kubelet/pods/08ce4326-4703-4061-aec4-17b4c6567785/volumes" Nov 03 01:20:24.787789 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:24.787732 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s"] Nov 03 01:20:24.788258 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:24.787989 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" podUID="8896996a-4482-4880-a631-3ab45648906b" containerName="maas-api" containerID="cri-o://6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f" gracePeriod=30 Nov 03 01:20:25.046176 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.046119 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 01:20:25.069581 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.069558 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-metrics-certs\") pod \"8896996a-4482-4880-a631-3ab45648906b\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " Nov 03 01:20:25.069704 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.069609 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8896996a-4482-4880-a631-3ab45648906b-kube-api-access-8gfrt\" (UniqueName: \"kubernetes.io/projected/8896996a-4482-4880-a631-3ab45648906b-kube-api-access-8gfrt\") pod \"8896996a-4482-4880-a631-3ab45648906b\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " Nov 03 01:20:25.069704 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.069639 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-maas-api-tls\") pod \"8896996a-4482-4880-a631-3ab45648906b\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " Nov 03 01:20:25.069704 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.069679 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8896996a-4482-4880-a631-3ab45648906b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8896996a-4482-4880-a631-3ab45648906b-openshift-service-ca\") pod \"8896996a-4482-4880-a631-3ab45648906b\" (UID: \"8896996a-4482-4880-a631-3ab45648906b\") " Nov 03 01:20:25.070056 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.070033 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8896996a-4482-4880-a631-3ab45648906b-openshift-service-ca" pod "8896996a-4482-4880-a631-3ab45648906b" (UID: "8896996a-4482-4880-a631-3ab45648906b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:25.071822 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.071789 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8896996a-4482-4880-a631-3ab45648906b-kube-api-access-8gfrt" pod "8896996a-4482-4880-a631-3ab45648906b" (UID: "8896996a-4482-4880-a631-3ab45648906b"). InnerVolumeSpecName "kube-api-access-8gfrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:25.071924 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.071870 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-maas-api-tls" pod "8896996a-4482-4880-a631-3ab45648906b" (UID: "8896996a-4482-4880-a631-3ab45648906b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:25.071924 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.071884 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-metrics-certs" pod "8896996a-4482-4880-a631-3ab45648906b" (UID: "8896996a-4482-4880-a631-3ab45648906b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:25.184695 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.184643 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:25.184695 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.184683 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gfrt\" (UniqueName: \"kubernetes.io/projected/8896996a-4482-4880-a631-3ab45648906b-kube-api-access-8gfrt\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:25.184695 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.184699 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8896996a-4482-4880-a631-3ab45648906b-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:25.185049 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.184713 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8896996a-4482-4880-a631-3ab45648906b-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:25.573730 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.573697 4304 generic.go:356] "Generic (PLEG): container finished" podID="8896996a-4482-4880-a631-3ab45648906b" containerID="6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f" exitCode=0 Nov 03 01:20:25.573941 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.573742 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" Nov 03 01:20:25.573941 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.573814 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" event={"ID":"8896996a-4482-4880-a631-3ab45648906b","Type":"ContainerDied","Data":"6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f"} Nov 03 01:20:25.573941 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.573875 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s" event={"ID":"8896996a-4482-4880-a631-3ab45648906b","Type":"ContainerDied","Data":"17414323b905963c2fb94d1c2a35868e980ae967373a90b52a66e6141f8570d9"} Nov 03 01:20:25.573941 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.573903 4304 scope.go:122] "RemoveContainer" containerID="6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f" Nov 03 01:20:25.584836 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.584818 4304 scope.go:122] "RemoveContainer" containerID="6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f" Nov 03 01:20:25.585093 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:20:25.585075 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f\": container with ID starting with 6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f not found: ID does not exist" containerID="6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f" Nov 03 01:20:25.585139 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.585101 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f"} err="failed to get container status \"6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f\": rpc error: code = NotFound desc = could not find container \"6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f\": container with ID starting with 6ed810041cfb305ef4b49d3aa408ea268ba283958a9289954bc84d523de0c44f not found: ID does not exist" Nov 03 01:20:25.736242 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.736209 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s"] Nov 03 01:20:25.739741 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:25.739713 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fce94b-676ddfdbb9-fhx8s"] Nov 03 01:20:26.837025 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:26.836991 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8896996a-4482-4880-a631-3ab45648906b" path="/var/lib/kubelet/pods/8896996a-4482-4880-a631-3ab45648906b/volumes" Nov 03 01:20:28.654029 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:20:28.653997 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:20:34.513114 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:34.513084 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff"] Nov 03 01:20:34.513526 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:34.513301 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" containerID="cri-o://4f0ea5c9633dece32dd8b8916509f32ec5df53d6c3994a4613c73aeaf3b4f80a" gracePeriod=30 Nov 03 01:20:39.663115 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.663058 4304 generic.go:356] "Generic (PLEG): container finished" podID="77ee583e-0050-4552-92de-d56e1305fb16" containerID="4f0ea5c9633dece32dd8b8916509f32ec5df53d6c3994a4613c73aeaf3b4f80a" exitCode=0 Nov 03 01:20:39.663595 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.663136 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" event={"ID":"77ee583e-0050-4552-92de-d56e1305fb16","Type":"ContainerDied","Data":"4f0ea5c9633dece32dd8b8916509f32ec5df53d6c3994a4613c73aeaf3b4f80a"} Nov 03 01:20:39.663595 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.663230 4304 scope.go:122] "RemoveContainer" containerID="9479afdd7a32216474f1808663825318404c8bb950c86e3563f3e65d4c26c32d" Nov 03 01:20:39.777467 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.777414 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 01:20:39.828873 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.828843 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-envoy\") pod \"77ee583e-0050-4552-92de-d56e1305fb16\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " Nov 03 01:20:39.828987 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.828884 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77ee583e-0050-4552-92de-d56e1305fb16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ee583e-0050-4552-92de-d56e1305fb16-istio-podinfo\") pod \"77ee583e-0050-4552-92de-d56e1305fb16\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " Nov 03 01:20:39.828987 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.828931 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-istio-token\" (UniqueName: \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-istio-token\") pod \"77ee583e-0050-4552-92de-d56e1305fb16\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " Nov 03 01:20:39.828987 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.828958 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ee583e-0050-4552-92de-d56e1305fb16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ee583e-0050-4552-92de-d56e1305fb16-istiod-ca-cert\") pod \"77ee583e-0050-4552-92de-d56e1305fb16\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " Nov 03 01:20:39.829157 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829001 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-kube-api-access-b6pc7\" (UniqueName: \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-kube-api-access-b6pc7\") pod \"77ee583e-0050-4552-92de-d56e1305fb16\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " Nov 03 01:20:39.829157 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829104 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-data\") pod \"77ee583e-0050-4552-92de-d56e1305fb16\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " Nov 03 01:20:39.829272 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829158 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-socket\") pod \"77ee583e-0050-4552-92de-d56e1305fb16\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " Nov 03 01:20:39.829272 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829191 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-credential-socket\") pod \"77ee583e-0050-4552-92de-d56e1305fb16\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " Nov 03 01:20:39.829272 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829233 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-certs\") pod \"77ee583e-0050-4552-92de-d56e1305fb16\" (UID: \"77ee583e-0050-4552-92de-d56e1305fb16\") " Nov 03 01:20:39.829450 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829414 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-data" pod "77ee583e-0050-4552-92de-d56e1305fb16" (UID: "77ee583e-0050-4552-92de-d56e1305fb16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:39.829511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829447 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ee583e-0050-4552-92de-d56e1305fb16-istiod-ca-cert" pod "77ee583e-0050-4552-92de-d56e1305fb16" (UID: "77ee583e-0050-4552-92de-d56e1305fb16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:39.829511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829440 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-socket" pod "77ee583e-0050-4552-92de-d56e1305fb16" (UID: "77ee583e-0050-4552-92de-d56e1305fb16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:39.829639 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829600 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-certs" pod "77ee583e-0050-4552-92de-d56e1305fb16" (UID: "77ee583e-0050-4552-92de-d56e1305fb16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:39.829851 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829830 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-credential-socket" pod "77ee583e-0050-4552-92de-d56e1305fb16" (UID: "77ee583e-0050-4552-92de-d56e1305fb16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:39.829938 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829865 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.829938 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829884 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ee583e-0050-4552-92de-d56e1305fb16-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.829938 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829900 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.829938 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.829918 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.831175 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.831149 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77ee583e-0050-4552-92de-d56e1305fb16-istio-podinfo" pod "77ee583e-0050-4552-92de-d56e1305fb16" (UID: "77ee583e-0050-4552-92de-d56e1305fb16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:20:39.831460 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.831441 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-envoy" pod "77ee583e-0050-4552-92de-d56e1305fb16" (UID: "77ee583e-0050-4552-92de-d56e1305fb16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:39.831725 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.831708 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-istio-token" pod "77ee583e-0050-4552-92de-d56e1305fb16" (UID: "77ee583e-0050-4552-92de-d56e1305fb16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:39.831768 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.831748 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-kube-api-access-b6pc7" pod "77ee583e-0050-4552-92de-d56e1305fb16" (UID: "77ee583e-0050-4552-92de-d56e1305fb16"). InnerVolumeSpecName "kube-api-access-b6pc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:39.931454 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.931429 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.931454 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.931450 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ee583e-0050-4552-92de-d56e1305fb16-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.931601 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.931459 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.931601 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.931467 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6pc7\" (UniqueName: \"kubernetes.io/projected/77ee583e-0050-4552-92de-d56e1305fb16-kube-api-access-b6pc7\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.931601 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:39.931477 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ee583e-0050-4552-92de-d56e1305fb16-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.670418 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:40.670387 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" Nov 03 01:20:40.670904 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:40.670420 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff" event={"ID":"77ee583e-0050-4552-92de-d56e1305fb16","Type":"ContainerDied","Data":"9465b9ccaf31796b6b0b342a0b32ca025c427e81155ad670cdb25c33b615b5e9"} Nov 03 01:20:40.670904 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:40.670495 4304 scope.go:122] "RemoveContainer" containerID="4f0ea5c9633dece32dd8b8916509f32ec5df53d6c3994a4613c73aeaf3b4f80a" Nov 03 01:20:40.882840 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:40.882802 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff"] Nov 03 01:20:40.932509 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:40.932443 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fce94b-openshift-default-5544b6d9b9-lqzff"] Nov 03 01:20:42.827982 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:42.827943 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ee583e-0050-4552-92de-d56e1305fb16" path="/var/lib/kubelet/pods/77ee583e-0050-4552-92de-d56e1305fb16/volumes" Nov 03 01:20:43.050251 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:43.050217 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs"] Nov 03 01:20:43.097290 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:43.097222 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq"] Nov 03 01:20:48.051088 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:48.051050 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" podUID="29794f37-0e98-48b3-aa42-dd12a31f9280" containerName="payload-processing" containerID="cri-o://21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89" gracePeriod=25 Nov 03 01:20:48.097843 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:48.097811 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" podUID="1a786707-9324-49d4-89dc-decd70a96ba5" containerName="payload-pre-processing" containerID="cri-o://d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0" gracePeriod=25 Nov 03 01:20:50.326819 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.326786 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57"] Nov 03 01:20:50.327248 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.326987 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" podUID="b83b4462-07b9-4c5a-8acb-8e7cb777d794" containerName="maas-api" containerID="cri-o://1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20" gracePeriod=30 Nov 03 01:20:50.588144 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.588086 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 01:20:50.628491 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.628462 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83b4462-07b9-4c5a-8acb-8e7cb777d794-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83b4462-07b9-4c5a-8acb-8e7cb777d794-openshift-service-ca\") pod \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " Nov 03 01:20:50.628636 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.628547 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83b4462-07b9-4c5a-8acb-8e7cb777d794-kube-api-access-4p9dv\" (UniqueName: \"kubernetes.io/projected/b83b4462-07b9-4c5a-8acb-8e7cb777d794-kube-api-access-4p9dv\") pod \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " Nov 03 01:20:50.628678 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.628649 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-metrics-certs\") pod \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " Nov 03 01:20:50.628760 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.628745 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-maas-api-tls\") pod \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\" (UID: \"b83b4462-07b9-4c5a-8acb-8e7cb777d794\") " Nov 03 01:20:50.628878 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.628858 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83b4462-07b9-4c5a-8acb-8e7cb777d794-openshift-service-ca" pod "b83b4462-07b9-4c5a-8acb-8e7cb777d794" (UID: "b83b4462-07b9-4c5a-8acb-8e7cb777d794"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:50.629107 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.629087 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83b4462-07b9-4c5a-8acb-8e7cb777d794-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:50.630723 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.630698 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-metrics-certs" pod "b83b4462-07b9-4c5a-8acb-8e7cb777d794" (UID: "b83b4462-07b9-4c5a-8acb-8e7cb777d794"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:50.630863 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.630748 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-maas-api-tls" pod "b83b4462-07b9-4c5a-8acb-8e7cb777d794" (UID: "b83b4462-07b9-4c5a-8acb-8e7cb777d794"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:50.630935 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.630908 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83b4462-07b9-4c5a-8acb-8e7cb777d794-kube-api-access-4p9dv" pod "b83b4462-07b9-4c5a-8acb-8e7cb777d794" (UID: "b83b4462-07b9-4c5a-8acb-8e7cb777d794"). InnerVolumeSpecName "kube-api-access-4p9dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:50.726690 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.726654 4304 generic.go:356] "Generic (PLEG): container finished" podID="b83b4462-07b9-4c5a-8acb-8e7cb777d794" containerID="1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20" exitCode=0 Nov 03 01:20:50.726904 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.726699 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" Nov 03 01:20:50.726904 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.726740 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" event={"ID":"b83b4462-07b9-4c5a-8acb-8e7cb777d794","Type":"ContainerDied","Data":"1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20"} Nov 03 01:20:50.726904 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.726815 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57" event={"ID":"b83b4462-07b9-4c5a-8acb-8e7cb777d794","Type":"ContainerDied","Data":"0a4c2dbace386376065085d225bd2f8a8e945ea16eca267ca215e3aa4ec25888"} Nov 03 01:20:50.726904 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.726850 4304 scope.go:122] "RemoveContainer" containerID="1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20" Nov 03 01:20:50.730312 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.730283 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:50.730416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.730317 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b83b4462-07b9-4c5a-8acb-8e7cb777d794-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:50.730416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.730334 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p9dv\" (UniqueName: \"kubernetes.io/projected/b83b4462-07b9-4c5a-8acb-8e7cb777d794-kube-api-access-4p9dv\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:50.739234 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.739211 4304 scope.go:122] "RemoveContainer" containerID="1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20" Nov 03 01:20:50.739499 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:20:50.739476 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20\": container with ID starting with 1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20 not found: ID does not exist" containerID="1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20" Nov 03 01:20:50.739570 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.739511 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20"} err="failed to get container status \"1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20\": rpc error: code = NotFound desc = could not find container \"1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20\": container with ID starting with 1816779bde786587d0ac2286d0ce64525019e64643b541b1970781471e848c20 not found: ID does not exist" Nov 03 01:20:50.889043 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.888971 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57"] Nov 03 01:20:50.891550 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:50.891527 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-531ee4-77c448cb6b-c6j57"] Nov 03 01:20:52.828618 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:52.828587 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83b4462-07b9-4c5a-8acb-8e7cb777d794" path="/var/lib/kubelet/pods/b83b4462-07b9-4c5a-8acb-8e7cb777d794/volumes" Nov 03 01:20:54.281100 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:54.281058 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" podUID="1a786707-9324-49d4-89dc-decd70a96ba5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:20:54.281528 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:20:54.281074 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" podUID="29794f37-0e98-48b3-aa42-dd12a31f9280" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:20:58.654880 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:20:58.654847 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:21:01.018629 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.018546 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t"] Nov 03 01:21:01.019123 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.018802 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" containerID="cri-o://d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a" gracePeriod=30 Nov 03 01:21:01.021216 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.021191 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc"] Nov 03 01:21:01.021435 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.021400 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" podUID="bff46dd8-5eb0-4114-8bdc-13964d0c0b23" containerName="maas-api" containerID="cri-o://4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc" gracePeriod=30 Nov 03 01:21:01.290042 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.290019 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 01:21:01.448134 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.448098 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-metrics-certs\") pod \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " Nov 03 01:21:01.448287 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.448150 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-openshift-service-ca\") pod \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " Nov 03 01:21:01.448287 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.448176 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-kube-api-access-drjkl\" (UniqueName: \"kubernetes.io/projected/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-kube-api-access-drjkl\") pod \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " Nov 03 01:21:01.448287 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.448233 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-maas-api-tls\") pod \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\" (UID: \"bff46dd8-5eb0-4114-8bdc-13964d0c0b23\") " Nov 03 01:21:01.448523 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.448499 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-openshift-service-ca" pod "bff46dd8-5eb0-4114-8bdc-13964d0c0b23" (UID: "bff46dd8-5eb0-4114-8bdc-13964d0c0b23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:01.450460 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.450437 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-metrics-certs" pod "bff46dd8-5eb0-4114-8bdc-13964d0c0b23" (UID: "bff46dd8-5eb0-4114-8bdc-13964d0c0b23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:01.450566 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.450524 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-maas-api-tls" pod "bff46dd8-5eb0-4114-8bdc-13964d0c0b23" (UID: "bff46dd8-5eb0-4114-8bdc-13964d0c0b23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:01.450641 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.450622 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-kube-api-access-drjkl" pod "bff46dd8-5eb0-4114-8bdc-13964d0c0b23" (UID: "bff46dd8-5eb0-4114-8bdc-13964d0c0b23"). InnerVolumeSpecName "kube-api-access-drjkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:01.549457 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.549382 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:01.549457 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.549409 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:01.549457 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.549421 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drjkl\" (UniqueName: \"kubernetes.io/projected/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-kube-api-access-drjkl\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:01.549457 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.549432 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bff46dd8-5eb0-4114-8bdc-13964d0c0b23-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:01.787585 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.787546 4304 generic.go:356] "Generic (PLEG): container finished" podID="bff46dd8-5eb0-4114-8bdc-13964d0c0b23" containerID="4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc" exitCode=0 Nov 03 01:21:01.787766 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.787595 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" Nov 03 01:21:01.787766 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.787627 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" event={"ID":"bff46dd8-5eb0-4114-8bdc-13964d0c0b23","Type":"ContainerDied","Data":"4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc"} Nov 03 01:21:01.787766 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.787682 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc" event={"ID":"bff46dd8-5eb0-4114-8bdc-13964d0c0b23","Type":"ContainerDied","Data":"4810dcd972d883edde543a1f6f331c8544e09a52e7c98c6bb96bed34208cafc2"} Nov 03 01:21:01.787766 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.787707 4304 scope.go:122] "RemoveContainer" containerID="4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc" Nov 03 01:21:01.797682 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.797665 4304 scope.go:122] "RemoveContainer" containerID="4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc" Nov 03 01:21:01.797955 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:21:01.797933 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc\": container with ID starting with 4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc not found: ID does not exist" containerID="4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc" Nov 03 01:21:01.798030 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.797962 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc"} err="failed to get container status \"4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc\": rpc error: code = NotFound desc = could not find container \"4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc\": container with ID starting with 4685992fc19267d7fde8d70b638b881d28a4ddec4d588bd206f19a21929b63fc not found: ID does not exist" Nov 03 01:21:01.948638 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.948614 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc"] Nov 03 01:21:01.952135 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:01.952113 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20f0a3-5f69bf6fff-85flc"] Nov 03 01:21:02.827412 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:02.827377 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff46dd8-5eb0-4114-8bdc-13964d0c0b23" path="/var/lib/kubelet/pods/bff46dd8-5eb0-4114-8bdc-13964d0c0b23/volumes" Nov 03 01:21:04.281890 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:04.281855 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" podUID="29794f37-0e98-48b3-aa42-dd12a31f9280" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:04.282310 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:04.282003 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" podUID="1a786707-9324-49d4-89dc-decd70a96ba5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:06.259707 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.259683 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 01:21:06.391485 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.391422 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-podinfo\") pod \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " Nov 03 01:21:06.391485 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.391469 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-kube-api-access-hrsvw\" (UniqueName: \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-kube-api-access-hrsvw\") pod \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " Nov 03 01:21:06.391643 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.391541 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-data\") pod \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " Nov 03 01:21:06.391643 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.391582 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istiod-ca-cert\") pod \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " Nov 03 01:21:06.391643 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.391605 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-credential-socket\") pod \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " Nov 03 01:21:06.391643 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.391641 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-token\" (UniqueName: \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-token\") pod \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " Nov 03 01:21:06.391874 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.391665 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-socket\") pod \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " Nov 03 01:21:06.391874 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.391706 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-envoy\") pod \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " Nov 03 01:21:06.391874 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.391737 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-certs\") pod \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\" (UID: \"e2a82d52-a209-48e0-bab6-e68c0c36e6e2\") " Nov 03 01:21:06.392026 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.391975 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-data" pod "e2a82d52-a209-48e0-bab6-e68c0c36e6e2" (UID: "e2a82d52-a209-48e0-bab6-e68c0c36e6e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:06.392026 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.391990 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-socket" pod "e2a82d52-a209-48e0-bab6-e68c0c36e6e2" (UID: "e2a82d52-a209-48e0-bab6-e68c0c36e6e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:06.392026 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.392010 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-credential-socket" pod "e2a82d52-a209-48e0-bab6-e68c0c36e6e2" (UID: "e2a82d52-a209-48e0-bab6-e68c0c36e6e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:06.392185 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.392059 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istiod-ca-cert" pod "e2a82d52-a209-48e0-bab6-e68c0c36e6e2" (UID: "e2a82d52-a209-48e0-bab6-e68c0c36e6e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:06.392439 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.392416 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-certs" pod "e2a82d52-a209-48e0-bab6-e68c0c36e6e2" (UID: "e2a82d52-a209-48e0-bab6-e68c0c36e6e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:06.392664 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.392635 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.392816 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.392676 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.392816 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.392693 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.392816 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.392705 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.392816 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.392718 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.393767 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.393744 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-podinfo" pod "e2a82d52-a209-48e0-bab6-e68c0c36e6e2" (UID: "e2a82d52-a209-48e0-bab6-e68c0c36e6e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:21:06.394099 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.394077 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-kube-api-access-hrsvw" pod "e2a82d52-a209-48e0-bab6-e68c0c36e6e2" (UID: "e2a82d52-a209-48e0-bab6-e68c0c36e6e2"). InnerVolumeSpecName "kube-api-access-hrsvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:06.394194 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.394169 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-token" pod "e2a82d52-a209-48e0-bab6-e68c0c36e6e2" (UID: "e2a82d52-a209-48e0-bab6-e68c0c36e6e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:06.394271 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.394251 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-envoy" pod "e2a82d52-a209-48e0-bab6-e68c0c36e6e2" (UID: "e2a82d52-a209-48e0-bab6-e68c0c36e6e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:06.494133 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.494098 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.494133 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.494133 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.494300 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.494149 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.494300 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.494165 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrsvw\" (UniqueName: \"kubernetes.io/projected/e2a82d52-a209-48e0-bab6-e68c0c36e6e2-kube-api-access-hrsvw\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.818378 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.818349 4304 generic.go:356] "Generic (PLEG): container finished" podID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerID="d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a" exitCode=0 Nov 03 01:21:06.818563 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.818395 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" Nov 03 01:21:06.818563 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.818440 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" event={"ID":"e2a82d52-a209-48e0-bab6-e68c0c36e6e2","Type":"ContainerDied","Data":"d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a"} Nov 03 01:21:06.818563 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.818496 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t" event={"ID":"e2a82d52-a209-48e0-bab6-e68c0c36e6e2","Type":"ContainerDied","Data":"fd075607721464a6e58cb5890a1d8bc0aa04e00ccbcb549ec6fafaf74185118a"} Nov 03 01:21:06.818563 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.818527 4304 scope.go:122] "RemoveContainer" containerID="d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a" Nov 03 01:21:06.829077 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.829055 4304 scope.go:122] "RemoveContainer" containerID="8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a" Nov 03 01:21:06.837127 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.837109 4304 scope.go:122] "RemoveContainer" containerID="d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a" Nov 03 01:21:06.837367 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:21:06.837348 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a\": container with ID starting with d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a not found: ID does not exist" containerID="d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a" Nov 03 01:21:06.837425 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.837374 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a"} err="failed to get container status \"d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a\": rpc error: code = NotFound desc = could not find container \"d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a\": container with ID starting with d83b8d06e459b9cd1dd0e14899c27bc56c10f9c7243e8d2bb149e52de6624a8a not found: ID does not exist" Nov 03 01:21:06.837425 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.837392 4304 scope.go:122] "RemoveContainer" containerID="8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a" Nov 03 01:21:06.837590 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:21:06.837575 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a\": container with ID starting with 8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a not found: ID does not exist" containerID="8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a" Nov 03 01:21:06.837626 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.837595 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a"} err="failed to get container status \"8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a\": rpc error: code = NotFound desc = could not find container \"8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a\": container with ID starting with 8d6800bb0c1f324bf94a40b2084541abfb468a04387b6052c63b0df0fedf921a not found: ID does not exist" Nov 03 01:21:06.983624 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:06.983601 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t"] Nov 03 01:21:07.032560 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:07.032535 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-xpp6t"] Nov 03 01:21:08.349864 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.349844 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 01:21:08.375924 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.375904 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 01:21:08.513258 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.513228 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a786707-9324-49d4-89dc-decd70a96ba5-kube-api-access-sc9l7\" (UniqueName: \"kubernetes.io/projected/1a786707-9324-49d4-89dc-decd70a96ba5-kube-api-access-sc9l7\") pod \"1a786707-9324-49d4-89dc-decd70a96ba5\" (UID: \"1a786707-9324-49d4-89dc-decd70a96ba5\") " Nov 03 01:21:08.513403 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.513274 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29794f37-0e98-48b3-aa42-dd12a31f9280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29794f37-0e98-48b3-aa42-dd12a31f9280-plugins-config-volume\") pod \"29794f37-0e98-48b3-aa42-dd12a31f9280\" (UID: \"29794f37-0e98-48b3-aa42-dd12a31f9280\") " Nov 03 01:21:08.513403 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.513298 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a786707-9324-49d4-89dc-decd70a96ba5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a786707-9324-49d4-89dc-decd70a96ba5-plugins-config-volume\") pod \"1a786707-9324-49d4-89dc-decd70a96ba5\" (UID: \"1a786707-9324-49d4-89dc-decd70a96ba5\") " Nov 03 01:21:08.513525 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.513427 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29794f37-0e98-48b3-aa42-dd12a31f9280-kube-api-access-vf9dd\" (UniqueName: \"kubernetes.io/projected/29794f37-0e98-48b3-aa42-dd12a31f9280-kube-api-access-vf9dd\") pod \"29794f37-0e98-48b3-aa42-dd12a31f9280\" (UID: \"29794f37-0e98-48b3-aa42-dd12a31f9280\") " Nov 03 01:21:08.513748 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.513724 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29794f37-0e98-48b3-aa42-dd12a31f9280-plugins-config-volume" pod "29794f37-0e98-48b3-aa42-dd12a31f9280" (UID: "29794f37-0e98-48b3-aa42-dd12a31f9280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:08.513858 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.513745 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a786707-9324-49d4-89dc-decd70a96ba5-plugins-config-volume" pod "1a786707-9324-49d4-89dc-decd70a96ba5" (UID: "1a786707-9324-49d4-89dc-decd70a96ba5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:08.515499 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.515481 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29794f37-0e98-48b3-aa42-dd12a31f9280-kube-api-access-vf9dd" pod "29794f37-0e98-48b3-aa42-dd12a31f9280" (UID: "29794f37-0e98-48b3-aa42-dd12a31f9280"). InnerVolumeSpecName "kube-api-access-vf9dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:08.515549 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.515498 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a786707-9324-49d4-89dc-decd70a96ba5-kube-api-access-sc9l7" pod "1a786707-9324-49d4-89dc-decd70a96ba5" (UID: "1a786707-9324-49d4-89dc-decd70a96ba5"). InnerVolumeSpecName "kube-api-access-sc9l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:08.615535 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.615445 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf9dd\" (UniqueName: \"kubernetes.io/projected/29794f37-0e98-48b3-aa42-dd12a31f9280-kube-api-access-vf9dd\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:08.615535 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.615479 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc9l7\" (UniqueName: \"kubernetes.io/projected/1a786707-9324-49d4-89dc-decd70a96ba5-kube-api-access-sc9l7\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:08.615535 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.615491 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29794f37-0e98-48b3-aa42-dd12a31f9280-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:08.615535 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.615505 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a786707-9324-49d4-89dc-decd70a96ba5-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:08.828244 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.828215 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" path="/var/lib/kubelet/pods/e2a82d52-a209-48e0-bab6-e68c0c36e6e2/volumes" Nov 03 01:21:08.829956 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.829924 4304 generic.go:356] "Generic (PLEG): container finished" podID="1a786707-9324-49d4-89dc-decd70a96ba5" containerID="d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0" exitCode=0 Nov 03 01:21:08.830113 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.829989 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" Nov 03 01:21:08.830113 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.830008 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" event={"ID":"1a786707-9324-49d4-89dc-decd70a96ba5","Type":"ContainerDied","Data":"d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0"} Nov 03 01:21:08.830113 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.830069 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq" event={"ID":"1a786707-9324-49d4-89dc-decd70a96ba5","Type":"ContainerDied","Data":"307fe70d7a9874adb432950a0b050a8a12ea7df038dac84fc2489d7a6adfba46"} Nov 03 01:21:08.830113 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.830096 4304 scope.go:122] "RemoveContainer" containerID="d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0" Nov 03 01:21:08.831797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.831707 4304 generic.go:356] "Generic (PLEG): container finished" podID="29794f37-0e98-48b3-aa42-dd12a31f9280" containerID="21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89" exitCode=0 Nov 03 01:21:08.831797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.831809 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" event={"ID":"29794f37-0e98-48b3-aa42-dd12a31f9280","Type":"ContainerDied","Data":"21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89"} Nov 03 01:21:08.832093 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.831769 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" Nov 03 01:21:08.832093 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.831858 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs" event={"ID":"29794f37-0e98-48b3-aa42-dd12a31f9280","Type":"ContainerDied","Data":"6011f7a87ec7aaa270f760b184610c87db9dae4d1b6b34365816a798dc887857"} Nov 03 01:21:08.845163 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.845148 4304 scope.go:122] "RemoveContainer" containerID="d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0" Nov 03 01:21:08.845484 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:21:08.845464 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0\": container with ID starting with d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0 not found: ID does not exist" containerID="d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0" Nov 03 01:21:08.845550 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.845491 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0"} err="failed to get container status \"d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0\": rpc error: code = NotFound desc = could not find container \"d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0\": container with ID starting with d7eb8f99e5ef2a596d0ce6c57d1f42f5f6be5f26fb0d70d538ec7a930daddbf0 not found: ID does not exist" Nov 03 01:21:08.845550 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.845515 4304 scope.go:122] "RemoveContainer" containerID="21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89" Nov 03 01:21:08.854025 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.854007 4304 scope.go:122] "RemoveContainer" containerID="21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89" Nov 03 01:21:08.854279 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:21:08.854261 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89\": container with ID starting with 21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89 not found: ID does not exist" containerID="21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89" Nov 03 01:21:08.854323 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.854285 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89"} err="failed to get container status \"21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89\": rpc error: code = NotFound desc = could not find container \"21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89\": container with ID starting with 21f2024d6b89e8564056889468addde7289116de8f3ad4afd727121964378c89 not found: ID does not exist" Nov 03 01:21:08.992859 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:08.992837 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq"] Nov 03 01:21:09.042224 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:09.042199 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20f0a3-5cf75c68b7-6fjqq"] Nov 03 01:21:09.102464 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:09.102443 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs"] Nov 03 01:21:09.106070 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:09.106041 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20f0a3-cb7db8567-b7xbs"] Nov 03 01:21:10.828766 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:10.828723 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a786707-9324-49d4-89dc-decd70a96ba5" path="/var/lib/kubelet/pods/1a786707-9324-49d4-89dc-decd70a96ba5/volumes" Nov 03 01:21:10.829387 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:10.829251 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29794f37-0e98-48b3-aa42-dd12a31f9280" path="/var/lib/kubelet/pods/29794f37-0e98-48b3-aa42-dd12a31f9280/volumes" Nov 03 01:21:18.795251 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:18.795225 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc"] Nov 03 01:21:21.675665 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:21.675633 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n"] Nov 03 01:21:21.676087 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:21.675893 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" podUID="63c8feee-4ebf-4910-9851-0744e618c916" containerName="maas-api" containerID="cri-o://647a0599096016f7ddf09971b7e2b83f04536af8cfc62e3e569873cb7ac97d8c" gracePeriod=30 Nov 03 01:21:21.900306 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:21.900273 4304 generic.go:356] "Generic (PLEG): container finished" podID="63c8feee-4ebf-4910-9851-0744e618c916" containerID="647a0599096016f7ddf09971b7e2b83f04536af8cfc62e3e569873cb7ac97d8c" exitCode=0 Nov 03 01:21:21.900499 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:21.900323 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" event={"ID":"63c8feee-4ebf-4910-9851-0744e618c916","Type":"ContainerDied","Data":"647a0599096016f7ddf09971b7e2b83f04536af8cfc62e3e569873cb7ac97d8c"} Nov 03 01:21:21.932175 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:21.932124 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 01:21:22.033845 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.033822 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c8feee-4ebf-4910-9851-0744e618c916-kube-api-access-vscjk\" (UniqueName: \"kubernetes.io/projected/63c8feee-4ebf-4910-9851-0744e618c916-kube-api-access-vscjk\") pod \"63c8feee-4ebf-4910-9851-0744e618c916\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " Nov 03 01:21:22.033992 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.033878 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-maas-api-tls\") pod \"63c8feee-4ebf-4910-9851-0744e618c916\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " Nov 03 01:21:22.033992 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.033962 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-metrics-certs\") pod \"63c8feee-4ebf-4910-9851-0744e618c916\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " Nov 03 01:21:22.034122 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.034108 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c8feee-4ebf-4910-9851-0744e618c916-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c8feee-4ebf-4910-9851-0744e618c916-openshift-service-ca\") pod \"63c8feee-4ebf-4910-9851-0744e618c916\" (UID: \"63c8feee-4ebf-4910-9851-0744e618c916\") " Nov 03 01:21:22.034619 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.034511 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c8feee-4ebf-4910-9851-0744e618c916-openshift-service-ca" pod "63c8feee-4ebf-4910-9851-0744e618c916" (UID: "63c8feee-4ebf-4910-9851-0744e618c916"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:22.036106 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.036082 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-metrics-certs" pod "63c8feee-4ebf-4910-9851-0744e618c916" (UID: "63c8feee-4ebf-4910-9851-0744e618c916"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:22.036215 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.036115 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-maas-api-tls" pod "63c8feee-4ebf-4910-9851-0744e618c916" (UID: "63c8feee-4ebf-4910-9851-0744e618c916"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:22.036215 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.036187 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c8feee-4ebf-4910-9851-0744e618c916-kube-api-access-vscjk" pod "63c8feee-4ebf-4910-9851-0744e618c916" (UID: "63c8feee-4ebf-4910-9851-0744e618c916"). InnerVolumeSpecName "kube-api-access-vscjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:22.135911 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.135879 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c8feee-4ebf-4910-9851-0744e618c916-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:22.135911 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.135906 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vscjk\" (UniqueName: \"kubernetes.io/projected/63c8feee-4ebf-4910-9851-0744e618c916-kube-api-access-vscjk\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:22.136093 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.135926 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:22.136093 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.135935 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c8feee-4ebf-4910-9851-0744e618c916-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:22.905930 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.905906 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" Nov 03 01:21:22.905930 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.905920 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n" event={"ID":"63c8feee-4ebf-4910-9851-0744e618c916","Type":"ContainerDied","Data":"70a642f68c42610cccdfa9b86b4a9b4b0262dfd8023dca882a81edb042a77fbf"} Nov 03 01:21:22.906376 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:22.905977 4304 scope.go:122] "RemoveContainer" containerID="647a0599096016f7ddf09971b7e2b83f04536af8cfc62e3e569873cb7ac97d8c" Nov 03 01:21:23.070799 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:23.070754 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n"] Nov 03 01:21:23.120491 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:23.120401 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b2e795-7b56d7d877-b486n"] Nov 03 01:21:24.831964 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:24.830365 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c8feee-4ebf-4910-9851-0744e618c916" path="/var/lib/kubelet/pods/63c8feee-4ebf-4910-9851-0744e618c916/volumes" Nov 03 01:21:28.654695 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:21:28.654657 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:21:33.487484 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:33.487452 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54"] Nov 03 01:21:33.489711 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:33.487646 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" podUID="080b642d-4c40-441d-9115-3c923bd13683" containerName="istio-proxy" containerID="cri-o://64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08" gracePeriod=30 Nov 03 01:21:33.815097 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:33.814977 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" podUID="aa4f830f-1dc3-4231-ab46-49e522d2e794" containerName="main" containerID="cri-o://cdbfae36adfeb86cebe7caebb2c501318f97ca3a62ddf6e9201e6bccea51acb8" gracePeriod=45 Nov 03 01:21:33.966814 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:33.966762 4304 generic.go:356] "Generic (PLEG): container finished" podID="aa4f830f-1dc3-4231-ab46-49e522d2e794" containerID="cdbfae36adfeb86cebe7caebb2c501318f97ca3a62ddf6e9201e6bccea51acb8" exitCode=0 Nov 03 01:21:33.966987 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:33.966812 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" event={"ID":"aa4f830f-1dc3-4231-ab46-49e522d2e794","Type":"ContainerDied","Data":"cdbfae36adfeb86cebe7caebb2c501318f97ca3a62ddf6e9201e6bccea51acb8"} Nov 03 01:21:34.076374 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.076318 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 01:21:34.144165 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.144134 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4f830f-1dc3-4231-ab46-49e522d2e794-kube-api-access-qmhsr\" (UniqueName: \"kubernetes.io/projected/aa4f830f-1dc3-4231-ab46-49e522d2e794-kube-api-access-qmhsr\") pod \"aa4f830f-1dc3-4231-ab46-49e522d2e794\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " Nov 03 01:21:34.144327 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.144203 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-model-cache\") pod \"aa4f830f-1dc3-4231-ab46-49e522d2e794\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " Nov 03 01:21:34.144327 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.144275 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-home\") pod \"aa4f830f-1dc3-4231-ab46-49e522d2e794\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " Nov 03 01:21:34.144327 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.144301 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-dshm\") pod \"aa4f830f-1dc3-4231-ab46-49e522d2e794\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " Nov 03 01:21:34.144495 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.144386 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4f830f-1dc3-4231-ab46-49e522d2e794-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f830f-1dc3-4231-ab46-49e522d2e794-tls-certs\") pod \"aa4f830f-1dc3-4231-ab46-49e522d2e794\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " Nov 03 01:21:34.144495 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.144463 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-tmp-dir\") pod \"aa4f830f-1dc3-4231-ab46-49e522d2e794\" (UID: \"aa4f830f-1dc3-4231-ab46-49e522d2e794\") " Nov 03 01:21:34.144495 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.144481 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-model-cache" pod "aa4f830f-1dc3-4231-ab46-49e522d2e794" (UID: "aa4f830f-1dc3-4231-ab46-49e522d2e794"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.144643 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.144496 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-home" pod "aa4f830f-1dc3-4231-ab46-49e522d2e794" (UID: "aa4f830f-1dc3-4231-ab46-49e522d2e794"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.144841 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.144756 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-tmp-dir" pod "aa4f830f-1dc3-4231-ab46-49e522d2e794" (UID: "aa4f830f-1dc3-4231-ab46-49e522d2e794"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.144841 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.144842 4304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-model-cache\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.144841 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.144861 4304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-home\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.146797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.146751 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4f830f-1dc3-4231-ab46-49e522d2e794-kube-api-access-qmhsr" pod "aa4f830f-1dc3-4231-ab46-49e522d2e794" (UID: "aa4f830f-1dc3-4231-ab46-49e522d2e794"). InnerVolumeSpecName "kube-api-access-qmhsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:34.147018 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.146997 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-dshm" pod "aa4f830f-1dc3-4231-ab46-49e522d2e794" (UID: "aa4f830f-1dc3-4231-ab46-49e522d2e794"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.147073 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.147023 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4f830f-1dc3-4231-ab46-49e522d2e794-tls-certs" pod "aa4f830f-1dc3-4231-ab46-49e522d2e794" (UID: "aa4f830f-1dc3-4231-ab46-49e522d2e794"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:34.245510 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.245475 4304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f830f-1dc3-4231-ab46-49e522d2e794-tls-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.245735 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.245708 4304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-tmp-dir\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.245891 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.245738 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmhsr\" (UniqueName: \"kubernetes.io/projected/aa4f830f-1dc3-4231-ab46-49e522d2e794-kube-api-access-qmhsr\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.245891 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.245753 4304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f830f-1dc3-4231-ab46-49e522d2e794-dshm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.973087 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.973056 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" Nov 03 01:21:34.973087 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.973073 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" event={"ID":"aa4f830f-1dc3-4231-ab46-49e522d2e794","Type":"ContainerDied","Data":"a38646422211c7a9069f3eaffbb46b0501faff1af5b50582b05297f3d8a4132c"} Nov 03 01:21:34.973555 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:34.973128 4304 scope.go:122] "RemoveContainer" containerID="cdbfae36adfeb86cebe7caebb2c501318f97ca3a62ddf6e9201e6bccea51acb8" Nov 03 01:21:35.054615 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:35.054584 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc" podUID="aa4f830f-1dc3-4231-ab46-49e522d2e794" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 01:21:35.135451 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:35.135383 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc"] Nov 03 01:21:35.137380 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:35.137361 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-w4-048135-kserve-f665cd4b-kt7zc"] Nov 03 01:21:36.831253 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:36.831223 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4f830f-1dc3-4231-ab46-49e522d2e794" path="/var/lib/kubelet/pods/aa4f830f-1dc3-4231-ab46-49e522d2e794/volumes" Nov 03 01:21:38.739048 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.739023 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:21:38.793652 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.793598 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-certs\") pod \"080b642d-4c40-441d-9115-3c923bd13683\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " Nov 03 01:21:38.793832 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.793652 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080b642d-4c40-441d-9115-3c923bd13683-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080b642d-4c40-441d-9115-3c923bd13683-istiod-ca-cert\") pod \"080b642d-4c40-441d-9115-3c923bd13683\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " Nov 03 01:21:38.793832 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.793688 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-envoy\") pod \"080b642d-4c40-441d-9115-3c923bd13683\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " Nov 03 01:21:38.793832 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.793714 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/080b642d-4c40-441d-9115-3c923bd13683-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080b642d-4c40-441d-9115-3c923bd13683-istio-podinfo\") pod \"080b642d-4c40-441d-9115-3c923bd13683\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " Nov 03 01:21:38.793832 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.793730 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-istio-token\" (UniqueName: \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-istio-token\") pod \"080b642d-4c40-441d-9115-3c923bd13683\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " Nov 03 01:21:38.793832 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.793767 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-data\") pod \"080b642d-4c40-441d-9115-3c923bd13683\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " Nov 03 01:21:38.793832 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.793814 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-socket\") pod \"080b642d-4c40-441d-9115-3c923bd13683\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " Nov 03 01:21:38.794182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.793850 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-kube-api-access-btsdp\" (UniqueName: \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-kube-api-access-btsdp\") pod \"080b642d-4c40-441d-9115-3c923bd13683\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " Nov 03 01:21:38.794182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.793892 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-credential-socket\") pod \"080b642d-4c40-441d-9115-3c923bd13683\" (UID: \"080b642d-4c40-441d-9115-3c923bd13683\") " Nov 03 01:21:38.794182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.793899 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-certs" pod "080b642d-4c40-441d-9115-3c923bd13683" (UID: "080b642d-4c40-441d-9115-3c923bd13683"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:38.794182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.794126 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-socket" pod "080b642d-4c40-441d-9115-3c923bd13683" (UID: "080b642d-4c40-441d-9115-3c923bd13683"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:38.794401 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.794191 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080b642d-4c40-441d-9115-3c923bd13683-istiod-ca-cert" pod "080b642d-4c40-441d-9115-3c923bd13683" (UID: "080b642d-4c40-441d-9115-3c923bd13683"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:38.794401 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.794316 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.794401 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.794332 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-credential-socket" pod "080b642d-4c40-441d-9115-3c923bd13683" (UID: "080b642d-4c40-441d-9115-3c923bd13683"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:38.794401 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.794349 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.794763 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.794736 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-data" pod "080b642d-4c40-441d-9115-3c923bd13683" (UID: "080b642d-4c40-441d-9115-3c923bd13683"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:38.796017 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.795990 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-istio-token" pod "080b642d-4c40-441d-9115-3c923bd13683" (UID: "080b642d-4c40-441d-9115-3c923bd13683"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:38.796106 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.796028 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-envoy" pod "080b642d-4c40-441d-9115-3c923bd13683" (UID: "080b642d-4c40-441d-9115-3c923bd13683"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:38.796106 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.796070 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/080b642d-4c40-441d-9115-3c923bd13683-istio-podinfo" pod "080b642d-4c40-441d-9115-3c923bd13683" (UID: "080b642d-4c40-441d-9115-3c923bd13683"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:21:38.796179 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.796109 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-kube-api-access-btsdp" pod "080b642d-4c40-441d-9115-3c923bd13683" (UID: "080b642d-4c40-441d-9115-3c923bd13683"). InnerVolumeSpecName "kube-api-access-btsdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:38.895027 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.895005 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080b642d-4c40-441d-9115-3c923bd13683-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.895027 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.895028 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.895027 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.895038 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080b642d-4c40-441d-9115-3c923bd13683-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.895237 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.895051 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.895237 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.895063 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.895237 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.895076 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btsdp\" (UniqueName: \"kubernetes.io/projected/080b642d-4c40-441d-9115-3c923bd13683-kube-api-access-btsdp\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.895237 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.895090 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080b642d-4c40-441d-9115-3c923bd13683-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.995697 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.995671 4304 generic.go:356] "Generic (PLEG): container finished" podID="080b642d-4c40-441d-9115-3c923bd13683" containerID="64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08" exitCode=0 Nov 03 01:21:38.995857 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.995728 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" Nov 03 01:21:38.995857 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.995739 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" event={"ID":"080b642d-4c40-441d-9115-3c923bd13683","Type":"ContainerDied","Data":"64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08"} Nov 03 01:21:38.995857 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.995804 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54" event={"ID":"080b642d-4c40-441d-9115-3c923bd13683","Type":"ContainerDied","Data":"623066d0d74ad0b302d5ee8668791bafcc2f9173314b2c9a54a3e84317f97906"} Nov 03 01:21:38.995857 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:38.995825 4304 scope.go:122] "RemoveContainer" containerID="64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08" Nov 03 01:21:39.004541 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:39.004524 4304 scope.go:122] "RemoveContainer" containerID="64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08" Nov 03 01:21:39.004822 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:21:39.004798 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08\": container with ID starting with 64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08 not found: ID does not exist" containerID="64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08" Nov 03 01:21:39.004911 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:39.004828 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08"} err="failed to get container status \"64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08\": rpc error: code = NotFound desc = could not find container \"64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08\": container with ID starting with 64f5f7f19082d27ed8f10423cd255d125e03e3b945a402f608b0618552f71e08 not found: ID does not exist" Nov 03 01:21:39.157439 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:39.157411 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54"] Nov 03 01:21:39.161062 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:39.161043 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-mgp54"] Nov 03 01:21:40.829729 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:40.829692 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080b642d-4c40-441d-9115-3c923bd13683" path="/var/lib/kubelet/pods/080b642d-4c40-441d-9115-3c923bd13683/volumes" Nov 03 01:21:42.490184 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.490147 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9"] Nov 03 01:21:42.490690 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.490384 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" podUID="ad821b12-e889-4259-894c-a7f4e294b33e" containerName="maas-api" containerID="cri-o://39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750" gracePeriod=30 Nov 03 01:21:42.746323 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.746265 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 01:21:42.829082 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.829055 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-maas-api-tls\") pod \"ad821b12-e889-4259-894c-a7f4e294b33e\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " Nov 03 01:21:42.829203 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.829104 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-metrics-certs\") pod \"ad821b12-e889-4259-894c-a7f4e294b33e\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " Nov 03 01:21:42.829203 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.829157 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad821b12-e889-4259-894c-a7f4e294b33e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad821b12-e889-4259-894c-a7f4e294b33e-openshift-service-ca\") pod \"ad821b12-e889-4259-894c-a7f4e294b33e\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " Nov 03 01:21:42.829319 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.829214 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad821b12-e889-4259-894c-a7f4e294b33e-kube-api-access-kst2b\" (UniqueName: \"kubernetes.io/projected/ad821b12-e889-4259-894c-a7f4e294b33e-kube-api-access-kst2b\") pod \"ad821b12-e889-4259-894c-a7f4e294b33e\" (UID: \"ad821b12-e889-4259-894c-a7f4e294b33e\") " Nov 03 01:21:42.829599 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.829569 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad821b12-e889-4259-894c-a7f4e294b33e-openshift-service-ca" pod "ad821b12-e889-4259-894c-a7f4e294b33e" (UID: "ad821b12-e889-4259-894c-a7f4e294b33e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:42.831263 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.831242 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-metrics-certs" pod "ad821b12-e889-4259-894c-a7f4e294b33e" (UID: "ad821b12-e889-4259-894c-a7f4e294b33e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:42.831339 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.831272 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-maas-api-tls" pod "ad821b12-e889-4259-894c-a7f4e294b33e" (UID: "ad821b12-e889-4259-894c-a7f4e294b33e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:42.831477 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.831464 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad821b12-e889-4259-894c-a7f4e294b33e-kube-api-access-kst2b" pod "ad821b12-e889-4259-894c-a7f4e294b33e" (UID: "ad821b12-e889-4259-894c-a7f4e294b33e"). InnerVolumeSpecName "kube-api-access-kst2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:42.930626 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.930601 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad821b12-e889-4259-894c-a7f4e294b33e-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:42.930626 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.930624 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kst2b\" (UniqueName: \"kubernetes.io/projected/ad821b12-e889-4259-894c-a7f4e294b33e-kube-api-access-kst2b\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:42.930807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.930635 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:42.930807 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:42.930644 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad821b12-e889-4259-894c-a7f4e294b33e-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:43.018746 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:43.018673 4304 generic.go:356] "Generic (PLEG): container finished" podID="ad821b12-e889-4259-894c-a7f4e294b33e" containerID="39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750" exitCode=0 Nov 03 01:21:43.018746 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:43.018724 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" Nov 03 01:21:43.018961 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:43.018746 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" event={"ID":"ad821b12-e889-4259-894c-a7f4e294b33e","Type":"ContainerDied","Data":"39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750"} Nov 03 01:21:43.018961 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:43.018816 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9" event={"ID":"ad821b12-e889-4259-894c-a7f4e294b33e","Type":"ContainerDied","Data":"1b86459ad32a7abad39fcbad432286787b9d9f8ff3cf0d44c3bb990dbec656d7"} Nov 03 01:21:43.018961 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:43.018851 4304 scope.go:122] "RemoveContainer" containerID="39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750" Nov 03 01:21:43.028174 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:43.028157 4304 scope.go:122] "RemoveContainer" containerID="39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750" Nov 03 01:21:43.028456 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:21:43.028437 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750\": container with ID starting with 39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750 not found: ID does not exist" containerID="39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750" Nov 03 01:21:43.028518 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:43.028464 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750"} err="failed to get container status \"39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750\": rpc error: code = NotFound desc = could not find container \"39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750\": container with ID starting with 39f31c64d811e01fb444c559ec9b52ce1225f7a962f46a62f58c11ebc9e2c750 not found: ID does not exist" Nov 03 01:21:43.179893 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:43.179857 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9"] Nov 03 01:21:43.185102 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:43.185077 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-904a34-55dff589f9-6g2v9"] Nov 03 01:21:43.766754 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:43.766725 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc"] Nov 03 01:21:43.767188 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:43.766929 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" podUID="19c95304-1ac6-4047-ad06-09946b5161dd" containerName="maas-api" containerID="cri-o://7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49" gracePeriod=30 Nov 03 01:21:44.011902 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.011881 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 01:21:44.025635 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.025576 4304 generic.go:356] "Generic (PLEG): container finished" podID="19c95304-1ac6-4047-ad06-09946b5161dd" containerID="7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49" exitCode=0 Nov 03 01:21:44.025635 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.025624 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" Nov 03 01:21:44.025763 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.025653 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" event={"ID":"19c95304-1ac6-4047-ad06-09946b5161dd","Type":"ContainerDied","Data":"7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49"} Nov 03 01:21:44.025763 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.025706 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" event={"ID":"19c95304-1ac6-4047-ad06-09946b5161dd","Type":"ContainerDied","Data":"246171495f9380fd3db854ed0d9a753f169c25b08328d0842ed5f4f976c96649"} Nov 03 01:21:44.025763 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.025731 4304 scope.go:122] "RemoveContainer" containerID="7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49" Nov 03 01:21:44.034973 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.034956 4304 scope.go:122] "RemoveContainer" containerID="7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49" Nov 03 01:21:44.035257 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:21:44.035237 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49\": container with ID starting with 7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49 not found: ID does not exist" containerID="7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49" Nov 03 01:21:44.035353 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.035264 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49"} err="failed to get container status \"7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49\": rpc error: code = NotFound desc = could not find container \"7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49\": container with ID starting with 7ba67cf57dbad308567b57ba3c53a78bb48bee19718361bfa05db7490974df49 not found: ID does not exist" Nov 03 01:21:44.142049 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.142021 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c95304-1ac6-4047-ad06-09946b5161dd-kube-api-access-8mqqm\" (UniqueName: \"kubernetes.io/projected/19c95304-1ac6-4047-ad06-09946b5161dd-kube-api-access-8mqqm\") pod \"19c95304-1ac6-4047-ad06-09946b5161dd\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " Nov 03 01:21:44.142211 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.142093 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-metrics-certs\") pod \"19c95304-1ac6-4047-ad06-09946b5161dd\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " Nov 03 01:21:44.142211 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.142110 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-maas-api-tls\") pod \"19c95304-1ac6-4047-ad06-09946b5161dd\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " Nov 03 01:21:44.142211 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.142135 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c95304-1ac6-4047-ad06-09946b5161dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c95304-1ac6-4047-ad06-09946b5161dd-openshift-service-ca\") pod \"19c95304-1ac6-4047-ad06-09946b5161dd\" (UID: \"19c95304-1ac6-4047-ad06-09946b5161dd\") " Nov 03 01:21:44.142625 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.142583 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c95304-1ac6-4047-ad06-09946b5161dd-openshift-service-ca" pod "19c95304-1ac6-4047-ad06-09946b5161dd" (UID: "19c95304-1ac6-4047-ad06-09946b5161dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:44.144226 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.144195 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c95304-1ac6-4047-ad06-09946b5161dd-kube-api-access-8mqqm" pod "19c95304-1ac6-4047-ad06-09946b5161dd" (UID: "19c95304-1ac6-4047-ad06-09946b5161dd"). InnerVolumeSpecName "kube-api-access-8mqqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:44.144356 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.144296 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-metrics-certs" pod "19c95304-1ac6-4047-ad06-09946b5161dd" (UID: "19c95304-1ac6-4047-ad06-09946b5161dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:44.144422 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.144354 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-maas-api-tls" pod "19c95304-1ac6-4047-ad06-09946b5161dd" (UID: "19c95304-1ac6-4047-ad06-09946b5161dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:44.243183 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.243157 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mqqm\" (UniqueName: \"kubernetes.io/projected/19c95304-1ac6-4047-ad06-09946b5161dd-kube-api-access-8mqqm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:44.243183 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.243178 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:44.243341 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.243189 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c95304-1ac6-4047-ad06-09946b5161dd-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:44.243341 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.243197 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c95304-1ac6-4047-ad06-09946b5161dd-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:44.444256 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.444221 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc"] Nov 03 01:21:44.447608 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.447584 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc"] Nov 03 01:21:44.828080 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.828036 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c95304-1ac6-4047-ad06-09946b5161dd" path="/var/lib/kubelet/pods/19c95304-1ac6-4047-ad06-09946b5161dd/volumes" Nov 03 01:21:44.828614 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:44.828593 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad821b12-e889-4259-894c-a7f4e294b33e" path="/var/lib/kubelet/pods/ad821b12-e889-4259-894c-a7f4e294b33e/volumes" Nov 03 01:21:46.942856 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:46.942814 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-048135-589dfb9796-7q8cc" podUID="19c95304-1ac6-4047-ad06-09946b5161dd" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.55:8443/health\": context deadline exceeded" Nov 03 01:21:58.509485 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:58.509456 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj"] Nov 03 01:21:58.509930 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:21:58.509697 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" containerID="cri-o://f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61" gracePeriod=30 Nov 03 01:21:58.665914 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:21:58.665868 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:22:03.764079 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.764057 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 01:22:03.917442 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.917414 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-kube-api-access-xkbvb\" (UniqueName: \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-kube-api-access-xkbvb\") pod \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " Nov 03 01:22:03.917579 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.917445 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-credential-socket\") pod \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " Nov 03 01:22:03.917579 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.917467 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-token\" (UniqueName: \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-token\") pod \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " Nov 03 01:22:03.917579 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.917497 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-envoy\") pod \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " Nov 03 01:22:03.917579 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.917515 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-podinfo\") pod \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " Nov 03 01:22:03.917873 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.917586 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istiod-ca-cert\") pod \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " Nov 03 01:22:03.917873 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.917610 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-certs\") pod \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " Nov 03 01:22:03.917873 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.917648 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-socket\") pod \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " Nov 03 01:22:03.917873 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.917699 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-data\") pod \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\" (UID: \"b31f9e36-ddf8-41a7-abcf-10d14702cb35\") " Nov 03 01:22:03.917873 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.917763 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-credential-socket" pod "b31f9e36-ddf8-41a7-abcf-10d14702cb35" (UID: "b31f9e36-ddf8-41a7-abcf-10d14702cb35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:03.918402 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.918051 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istiod-ca-cert" pod "b31f9e36-ddf8-41a7-abcf-10d14702cb35" (UID: "b31f9e36-ddf8-41a7-abcf-10d14702cb35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:03.918402 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.918066 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:03.918402 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.918144 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-certs" pod "b31f9e36-ddf8-41a7-abcf-10d14702cb35" (UID: "b31f9e36-ddf8-41a7-abcf-10d14702cb35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:03.918402 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.918209 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-data" pod "b31f9e36-ddf8-41a7-abcf-10d14702cb35" (UID: "b31f9e36-ddf8-41a7-abcf-10d14702cb35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:03.918632 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.918436 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-socket" pod "b31f9e36-ddf8-41a7-abcf-10d14702cb35" (UID: "b31f9e36-ddf8-41a7-abcf-10d14702cb35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:03.919762 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.919740 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-envoy" pod "b31f9e36-ddf8-41a7-abcf-10d14702cb35" (UID: "b31f9e36-ddf8-41a7-abcf-10d14702cb35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:03.919762 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.919756 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-kube-api-access-xkbvb" pod "b31f9e36-ddf8-41a7-abcf-10d14702cb35" (UID: "b31f9e36-ddf8-41a7-abcf-10d14702cb35"). InnerVolumeSpecName "kube-api-access-xkbvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:03.919917 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.919797 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-token" pod "b31f9e36-ddf8-41a7-abcf-10d14702cb35" (UID: "b31f9e36-ddf8-41a7-abcf-10d14702cb35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:03.919969 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:03.919950 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-podinfo" pod "b31f9e36-ddf8-41a7-abcf-10d14702cb35" (UID: "b31f9e36-ddf8-41a7-abcf-10d14702cb35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:22:04.019211 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.019182 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:04.019211 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.019207 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:04.019211 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.019217 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:04.019469 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.019225 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:04.019469 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.019233 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:04.019469 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.019240 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:04.019469 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.019249 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkbvb\" (UniqueName: \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-kube-api-access-xkbvb\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:04.019469 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.019269 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f9e36-ddf8-41a7-abcf-10d14702cb35-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:04.124798 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.124754 4304 generic.go:356] "Generic (PLEG): container finished" podID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerID="f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61" exitCode=0 Nov 03 01:22:04.124927 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.124829 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" Nov 03 01:22:04.124927 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.124840 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" event={"ID":"b31f9e36-ddf8-41a7-abcf-10d14702cb35","Type":"ContainerDied","Data":"f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61"} Nov 03 01:22:04.124927 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.124901 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj" event={"ID":"b31f9e36-ddf8-41a7-abcf-10d14702cb35","Type":"ContainerDied","Data":"5b3de6b4de2e96f0ef8a025b9d1a968d5ae93123e3894920a9b3edfb44ca140c"} Nov 03 01:22:04.125038 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.124930 4304 scope.go:122] "RemoveContainer" containerID="f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61" Nov 03 01:22:04.134954 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.134937 4304 scope.go:122] "RemoveContainer" containerID="fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0" Nov 03 01:22:04.143091 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.143076 4304 scope.go:122] "RemoveContainer" containerID="f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61" Nov 03 01:22:04.143371 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:22:04.143350 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61\": container with ID starting with f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61 not found: ID does not exist" containerID="f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61" Nov 03 01:22:04.143426 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.143380 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61"} err="failed to get container status \"f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61\": rpc error: code = NotFound desc = could not find container \"f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61\": container with ID starting with f2ebab4e54a34c4a77f9d07cea9d88f959bcb3483bd884eb2cb3d71465197a61 not found: ID does not exist" Nov 03 01:22:04.143426 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.143398 4304 scope.go:122] "RemoveContainer" containerID="fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0" Nov 03 01:22:04.143613 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:22:04.143594 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0\": container with ID starting with fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0 not found: ID does not exist" containerID="fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0" Nov 03 01:22:04.143653 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.143619 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0"} err="failed to get container status \"fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0\": rpc error: code = NotFound desc = could not find container \"fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0\": container with ID starting with fa17e89218e63627dbd3c7019d9f59437db1f1304806571b3681d8f326f5a2c0 not found: ID does not exist" Nov 03 01:22:04.287986 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.287930 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj"] Nov 03 01:22:04.291201 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.291182 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-9xfdj"] Nov 03 01:22:04.827407 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:04.827381 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" path="/var/lib/kubelet/pods/b31f9e36-ddf8-41a7-abcf-10d14702cb35/volumes" Nov 03 01:22:07.658744 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:07.658713 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz"] Nov 03 01:22:22.680188 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.680102 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" podUID="2fb1752b-050f-4042-8650-0be7a24c004e" containerName="main" containerID="cri-o://06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef" gracePeriod=45 Nov 03 01:22:22.928404 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.928373 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 01:22:22.977001 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.976945 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-home\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-home\") pod \"2fb1752b-050f-4042-8650-0be7a24c004e\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " Nov 03 01:22:22.977001 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.977001 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb1752b-050f-4042-8650-0be7a24c004e-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb1752b-050f-4042-8650-0be7a24c004e-tls-certs\") pod \"2fb1752b-050f-4042-8650-0be7a24c004e\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " Nov 03 01:22:22.977203 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.977020 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-dshm\") pod \"2fb1752b-050f-4042-8650-0be7a24c004e\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " Nov 03 01:22:22.977203 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.977061 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-model-cache\") pod \"2fb1752b-050f-4042-8650-0be7a24c004e\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " Nov 03 01:22:22.977203 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.977128 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb1752b-050f-4042-8650-0be7a24c004e-kube-api-access-wtdzs\" (UniqueName: \"kubernetes.io/projected/2fb1752b-050f-4042-8650-0be7a24c004e-kube-api-access-wtdzs\") pod \"2fb1752b-050f-4042-8650-0be7a24c004e\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " Nov 03 01:22:22.977203 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.977178 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-tmp-dir\") pod \"2fb1752b-050f-4042-8650-0be7a24c004e\" (UID: \"2fb1752b-050f-4042-8650-0be7a24c004e\") " Nov 03 01:22:22.977427 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.977212 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-home" pod "2fb1752b-050f-4042-8650-0be7a24c004e" (UID: "2fb1752b-050f-4042-8650-0be7a24c004e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:22.977427 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.977383 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-model-cache" pod "2fb1752b-050f-4042-8650-0be7a24c004e" (UID: "2fb1752b-050f-4042-8650-0be7a24c004e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:22.977511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.977476 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-tmp-dir" pod "2fb1752b-050f-4042-8650-0be7a24c004e" (UID: "2fb1752b-050f-4042-8650-0be7a24c004e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:22.977511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.977495 4304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-home\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:22.977587 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.977513 4304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-model-cache\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:22.979960 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.979408 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb1752b-050f-4042-8650-0be7a24c004e-tls-certs" pod "2fb1752b-050f-4042-8650-0be7a24c004e" (UID: "2fb1752b-050f-4042-8650-0be7a24c004e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:22:22.979960 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.979477 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-dshm" pod "2fb1752b-050f-4042-8650-0be7a24c004e" (UID: "2fb1752b-050f-4042-8650-0be7a24c004e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:22.979960 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:22.979490 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb1752b-050f-4042-8650-0be7a24c004e-kube-api-access-wtdzs" pod "2fb1752b-050f-4042-8650-0be7a24c004e" (UID: "2fb1752b-050f-4042-8650-0be7a24c004e"). InnerVolumeSpecName "kube-api-access-wtdzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:23.078527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.078501 4304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb1752b-050f-4042-8650-0be7a24c004e-tls-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.078527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.078525 4304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-dshm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.078710 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.078536 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtdzs\" (UniqueName: \"kubernetes.io/projected/2fb1752b-050f-4042-8650-0be7a24c004e-kube-api-access-wtdzs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.078710 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.078545 4304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb1752b-050f-4042-8650-0be7a24c004e-tmp-dir\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.219308 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.219274 4304 generic.go:356] "Generic (PLEG): container finished" podID="2fb1752b-050f-4042-8650-0be7a24c004e" containerID="06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef" exitCode=0 Nov 03 01:22:23.219468 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.219321 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" Nov 03 01:22:23.219468 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.219358 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" event={"ID":"2fb1752b-050f-4042-8650-0be7a24c004e","Type":"ContainerDied","Data":"06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef"} Nov 03 01:22:23.219468 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.219402 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz" event={"ID":"2fb1752b-050f-4042-8650-0be7a24c004e","Type":"ContainerDied","Data":"f365b4d22a25c802b8d181760a68b5b7369356d10f975390310defdbcfb9ebb4"} Nov 03 01:22:23.219468 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.219424 4304 scope.go:122] "RemoveContainer" containerID="06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef" Nov 03 01:22:23.228862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.228840 4304 scope.go:122] "RemoveContainer" containerID="06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef" Nov 03 01:22:23.229141 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:22:23.229114 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef\": container with ID starting with 06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef not found: ID does not exist" containerID="06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef" Nov 03 01:22:23.229207 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.229150 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef"} err="failed to get container status \"06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef\": rpc error: code = NotFound desc = could not find container \"06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef\": container with ID starting with 06a6aa748328eef37708ed74a0e005a26eeba3297630cd43397ad321e5e78bef not found: ID does not exist" Nov 03 01:22:23.380970 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.380935 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz"] Nov 03 01:22:23.384690 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:23.384666 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-w1-e28e1a-kserve-9b8dcd64b-bvbmz"] Nov 03 01:22:24.827415 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:24.827380 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb1752b-050f-4042-8650-0be7a24c004e" path="/var/lib/kubelet/pods/2fb1752b-050f-4042-8650-0be7a24c004e/volumes" Nov 03 01:22:28.654513 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:22:28.654479 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:22:39.050650 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.050556 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n"] Nov 03 01:22:39.051124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051049 4304 cpu_manager.go:414] "containerMap: removing container" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" Nov 03 01:22:39.051124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051066 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" Nov 03 01:22:39.051124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051082 4304 cpu_manager.go:414] "containerMap: removing container" podUID="1a786707-9324-49d4-89dc-decd70a96ba5" containerName="payload-pre-processing" Nov 03 01:22:39.051124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051089 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a786707-9324-49d4-89dc-decd70a96ba5" containerName="payload-pre-processing" Nov 03 01:22:39.051124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051099 4304 cpu_manager.go:414] "containerMap: removing container" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" Nov 03 01:22:39.051124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051106 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" Nov 03 01:22:39.051124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051126 4304 cpu_manager.go:414] "containerMap: removing container" podUID="8896996a-4482-4880-a631-3ab45648906b" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051133 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8896996a-4482-4880-a631-3ab45648906b" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051142 4304 cpu_manager.go:414] "containerMap: removing container" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051149 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051157 4304 cpu_manager.go:414] "containerMap: removing container" podUID="26d57814-f91a-423c-ab38-5cb55e4d77a0" containerName="main" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051164 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d57814-f91a-423c-ab38-5cb55e4d77a0" containerName="main" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051177 4304 cpu_manager.go:414] "containerMap: removing container" podUID="30838ca2-820a-4d11-8ce5-5dfd2d579bf8" containerName="main" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051185 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30838ca2-820a-4d11-8ce5-5dfd2d579bf8" containerName="main" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051200 4304 cpu_manager.go:414] "containerMap: removing container" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051207 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051216 4304 cpu_manager.go:414] "containerMap: removing container" podUID="63c8feee-4ebf-4910-9851-0744e618c916" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051223 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c8feee-4ebf-4910-9851-0744e618c916" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051244 4304 cpu_manager.go:414] "containerMap: removing container" podUID="29794f37-0e98-48b3-aa42-dd12a31f9280" containerName="payload-processing" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051251 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29794f37-0e98-48b3-aa42-dd12a31f9280" containerName="payload-processing" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051260 4304 cpu_manager.go:414] "containerMap: removing container" podUID="19c95304-1ac6-4047-ad06-09946b5161dd" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051266 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c95304-1ac6-4047-ad06-09946b5161dd" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051277 4304 cpu_manager.go:414] "containerMap: removing container" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051283 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051292 4304 cpu_manager.go:414] "containerMap: removing container" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051298 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051319 4304 cpu_manager.go:414] "containerMap: removing container" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051327 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051335 4304 cpu_manager.go:414] "containerMap: removing container" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051345 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051360 4304 cpu_manager.go:414] "containerMap: removing container" podUID="ad821b12-e889-4259-894c-a7f4e294b33e" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051372 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad821b12-e889-4259-894c-a7f4e294b33e" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051381 4304 cpu_manager.go:414] "containerMap: removing container" podUID="10cba01f-ff75-478e-b1db-09d49dd314da" containerName="payload-processing" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051389 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cba01f-ff75-478e-b1db-09d49dd314da" containerName="payload-processing" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051399 4304 cpu_manager.go:414] "containerMap: removing container" podUID="321e724f-037e-4159-ab48-4b852a01f18f" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051405 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321e724f-037e-4159-ab48-4b852a01f18f" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051413 4304 cpu_manager.go:414] "containerMap: removing container" podUID="b83b4462-07b9-4c5a-8acb-8e7cb777d794" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051420 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83b4462-07b9-4c5a-8acb-8e7cb777d794" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051430 4304 cpu_manager.go:414] "containerMap: removing container" podUID="aa4f830f-1dc3-4231-ab46-49e522d2e794" containerName="main" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051437 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4f830f-1dc3-4231-ab46-49e522d2e794" containerName="main" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051447 4304 cpu_manager.go:414] "containerMap: removing container" podUID="2fb1752b-050f-4042-8650-0be7a24c004e" containerName="main" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051454 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb1752b-050f-4042-8650-0be7a24c004e" containerName="main" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051463 4304 cpu_manager.go:414] "containerMap: removing container" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051470 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051486 4304 cpu_manager.go:414] "containerMap: removing container" podUID="0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1" containerName="payload-pre-processing" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051493 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1" containerName="payload-pre-processing" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051504 4304 cpu_manager.go:414] "containerMap: removing container" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051510 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051518 4304 cpu_manager.go:414] "containerMap: removing container" podUID="bff46dd8-5eb0-4114-8bdc-13964d0c0b23" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051526 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff46dd8-5eb0-4114-8bdc-13964d0c0b23" containerName="maas-api" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051539 4304 cpu_manager.go:414] "containerMap: removing container" podUID="082c2451-3bb7-4b3a-82c0-f8ea634adcdb" containerName="main" Nov 03 01:22:39.051527 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051546 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082c2451-3bb7-4b3a-82c0-f8ea634adcdb" containerName="main" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051558 4304 cpu_manager.go:414] "containerMap: removing container" podUID="08ce4326-4703-4061-aec4-17b4c6567785" containerName="main" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051564 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ce4326-4703-4061-aec4-17b4c6567785" containerName="main" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051576 4304 cpu_manager.go:414] "containerMap: removing container" podUID="080b642d-4c40-441d-9115-3c923bd13683" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051582 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080b642d-4c40-441d-9115-3c923bd13683" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051671 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051683 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c8feee-4ebf-4910-9851-0744e618c916" containerName="maas-api" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051692 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="29794f37-0e98-48b3-aa42-dd12a31f9280" containerName="payload-processing" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051699 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c95304-1ac6-4047-ad06-09946b5161dd" containerName="maas-api" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051707 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051715 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1c30b9-b6df-411c-8290-ff0a313629f7" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051726 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31f9e36-ddf8-41a7-abcf-10d14702cb35" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051734 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051740 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83b4462-07b9-4c5a-8acb-8e7cb777d794" containerName="maas-api" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051749 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad821b12-e889-4259-894c-a7f4e294b33e" containerName="maas-api" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051756 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cba01f-ff75-478e-b1db-09d49dd314da" containerName="payload-processing" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051764 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="321e724f-037e-4159-ab48-4b852a01f18f" containerName="maas-api" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051789 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4f830f-1dc3-4231-ab46-49e522d2e794" containerName="main" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051796 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb1752b-050f-4042-8650-0be7a24c004e" containerName="main" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051805 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051813 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb0a4ee-72bb-46a1-b1f7-3038127e5ef1" containerName="payload-pre-processing" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051821 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ee583e-0050-4552-92de-d56e1305fb16" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051830 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff46dd8-5eb0-4114-8bdc-13964d0c0b23" containerName="maas-api" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051837 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051844 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="082c2451-3bb7-4b3a-82c0-f8ea634adcdb" containerName="main" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051852 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ce4326-4703-4061-aec4-17b4c6567785" containerName="main" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051860 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="080b642d-4c40-441d-9115-3c923bd13683" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051870 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051879 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a786707-9324-49d4-89dc-decd70a96ba5" containerName="payload-pre-processing" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051887 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="262aac2d-8690-4618-89f9-32c9e2eb8226" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051897 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8896996a-4482-4880-a631-3ab45648906b" containerName="maas-api" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051905 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a82d52-a209-48e0-bab6-e68c0c36e6e2" containerName="istio-proxy" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051912 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d57814-f91a-423c-ab38-5cb55e4d77a0" containerName="main" Nov 03 01:22:39.053492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.051921 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="30838ca2-820a-4d11-8ce5-5dfd2d579bf8" containerName="main" Nov 03 01:22:39.055227 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.055209 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.107013 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.106990 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e28e1a-openshift-default-dockercfg-4zwkf\"" Nov 03 01:22:39.108267 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.108248 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.108331 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.108294 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.108331 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.108321 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.108437 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.108377 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.108481 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.108467 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.108514 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.108494 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.108545 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.108510 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.108581 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.108546 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.108638 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.108627 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7m78\" (UniqueName: \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-kube-api-access-q7m78\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.159002 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.158981 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n"] Nov 03 01:22:39.209979 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.209956 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7m78\" (UniqueName: \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-kube-api-access-q7m78\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210117 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.209998 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210117 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.210031 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210117 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.210051 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210117 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.210069 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210117 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.210089 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210117 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.210105 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210117 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.210120 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210438 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.210143 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.210489 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210576 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.210532 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210630 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.210578 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.210781 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.210756 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.211149 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.211130 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.212483 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.212462 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.212798 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.212754 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.269703 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.269674 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.269851 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.269793 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7m78\" (UniqueName: \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-kube-api-access-q7m78\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.367894 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.367826 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:39.790088 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.790064 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n"] Nov 03 01:22:39.792095 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:22:39.792066 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab4e1dd_5ed3_4ea0_b315_4ca2001f73a9.slice/crio-c5a0e609aa43baf428329ae2bfb01335921fd745416d6f461fa1f1f27e7a7959 WatchSource:0}: Error finding container c5a0e609aa43baf428329ae2bfb01335921fd745416d6f461fa1f1f27e7a7959: Status 404 returned error can't find the container with id c5a0e609aa43baf428329ae2bfb01335921fd745416d6f461fa1f1f27e7a7959 Nov 03 01:22:39.794163 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.794132 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:22:39.794240 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.794194 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:22:39.794240 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:39.794224 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:22:40.305568 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:40.305533 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" event={"ID":"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9","Type":"ContainerStarted","Data":"62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e"} Nov 03 01:22:40.306030 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:40.305581 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" event={"ID":"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9","Type":"ContainerStarted","Data":"c5a0e609aa43baf428329ae2bfb01335921fd745416d6f461fa1f1f27e7a7959"} Nov 03 01:22:40.368513 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:40.368484 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:40.369992 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:40.369965 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:22:40.370093 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:40.370010 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 01:22:40.440842 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:40.422391 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" podStartSLOduration=1.422374693 podStartE2EDuration="1.422374693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:40.420376336 +0000 UTC m=+4643.703934009" watchObservedRunningTime="2026-11-03 01:22:40.422374693 +0000 UTC m=+4643.705932367" Nov 03 01:22:41.368284 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:41.368251 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:22:41.368679 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:41.368303 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 01:22:42.368607 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:42.368568 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:22:42.369029 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:42.368632 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 01:22:43.368669 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:43.368633 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:22:43.369117 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:43.368689 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 01:22:44.368956 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:44.368923 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:22:44.369335 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:44.368975 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 01:22:45.368492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:45.368457 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:22:45.368708 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:45.368531 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 01:22:46.368537 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:46.368499 4304 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:22:46.368989 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:46.368565 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 01:22:47.372461 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:47.372431 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:47.372850 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:47.372713 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:47.373430 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:47.373414 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:48.554176 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:48.554143 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n"] Nov 03 01:22:49.370984 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:49.370954 4304 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" secret="" err="secret \"e2e-worker-w1-e28e1a-openshift-default-dockercfg-4zwkf\" not found" Nov 03 01:22:50.360869 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:50.360842 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xlr9b_f9b55548-d7c6-4a1f-971b-0b679da43de1/manager/0.log" Nov 03 01:22:50.375539 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:50.375507 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerName="istio-proxy" containerID="cri-o://62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e" gracePeriod=30 Nov 03 01:22:50.674124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:50.674096 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6bknt_1c5959e1-69b6-46e0-94e3-3c6c2e0de70b/manager/0.log" Nov 03 01:22:50.897918 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:50.897890 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-24fmd_71adc051-8d66-4db6-a823-2d912ff1d9ad/kuadrant-console-plugin/0.log" Nov 03 01:22:53.018244 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:53.018216 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-kg9lb_98df725b-22cd-4d9c-b61f-fcaae9c848dc/manager/0.log" Nov 03 01:22:54.350751 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:54.350719 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j"] Nov 03 01:22:54.351336 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:54.350919 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" containerID="cri-o://30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde" gracePeriod=30 Nov 03 01:22:54.647651 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:54.647624 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rttv9_ff259ff5-f5a0-47f7-8a8e-f2f6125748e5/manager/0.log" Nov 03 01:22:54.945834 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:54.945725 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:22:54.945834 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:54.945795 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:22:55.618641 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.618611 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:55.778610 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.778577 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istiod-ca-cert\") pod \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " Nov 03 01:22:55.778833 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.778620 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-data\") pod \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " Nov 03 01:22:55.778833 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.778709 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-certs\") pod \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " Nov 03 01:22:55.778833 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.778764 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-token\" (UniqueName: \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-token\") pod \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " Nov 03 01:22:55.778833 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.778828 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-envoy\") pod \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " Nov 03 01:22:55.779077 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.778862 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-kube-api-access-q7m78\" (UniqueName: \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-kube-api-access-q7m78\") pod \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " Nov 03 01:22:55.779077 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.778893 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-socket\") pod \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " Nov 03 01:22:55.779077 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.778937 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-credential-socket\") pod \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " Nov 03 01:22:55.779077 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.778977 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-podinfo\") pod \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\" (UID: \"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9\") " Nov 03 01:22:55.779077 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.779009 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-data" pod "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" (UID: "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:55.779389 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.779080 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istiod-ca-cert" pod "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" (UID: "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:55.779389 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.779056 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-certs" pod "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" (UID: "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:55.779389 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.779253 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-socket" pod "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" (UID: "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:55.779389 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.779317 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-credential-socket" pod "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" (UID: "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:55.779389 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.779378 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:55.779647 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.779399 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:55.779647 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.779413 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:55.779647 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.779427 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:55.781353 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.781325 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-envoy" pod "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" (UID: "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:55.781579 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.781562 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-kube-api-access-q7m78" pod "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" (UID: "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9"). InnerVolumeSpecName "kube-api-access-q7m78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:55.781786 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.781756 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-token" pod "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" (UID: "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:55.781857 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.781763 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-podinfo" pod "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" (UID: "bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:22:55.880275 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.880245 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:55.880275 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.880270 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:55.880447 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.880285 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:55.880447 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.880298 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7m78\" (UniqueName: \"kubernetes.io/projected/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-kube-api-access-q7m78\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:55.880447 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:55.880311 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:56.031596 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:56.031533 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dzrml_22dc8845-4b7d-41fb-9c91-cebde8326e19/manager/0.log" Nov 03 01:22:56.406921 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:56.406893 4304 generic.go:356] "Generic (PLEG): container finished" podID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerID="62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e" exitCode=0 Nov 03 01:22:56.407111 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:56.406950 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" Nov 03 01:22:56.407111 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:56.406975 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" event={"ID":"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9","Type":"ContainerDied","Data":"62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e"} Nov 03 01:22:56.407111 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:56.407015 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n" event={"ID":"bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9","Type":"ContainerDied","Data":"c5a0e609aa43baf428329ae2bfb01335921fd745416d6f461fa1f1f27e7a7959"} Nov 03 01:22:56.407111 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:56.407035 4304 scope.go:122] "RemoveContainer" containerID="62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e" Nov 03 01:22:56.416633 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:56.416616 4304 scope.go:122] "RemoveContainer" containerID="62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e" Nov 03 01:22:56.416979 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:22:56.416960 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e\": container with ID starting with 62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e not found: ID does not exist" containerID="62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e" Nov 03 01:22:56.417035 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:56.416988 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e"} err="failed to get container status \"62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e\": rpc error: code = NotFound desc = could not find container \"62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e\": container with ID starting with 62ac09b8a495c0180224cdca4607421fbcea2460a8321156bfa2a39a4f41e02e not found: ID does not exist" Nov 03 01:22:56.618144 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:56.618117 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n"] Nov 03 01:22:56.667522 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:56.667453 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-tgx9n"] Nov 03 01:22:56.829214 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:56.829157 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" path="/var/lib/kubelet/pods/bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9/volumes" Nov 03 01:22:58.656277 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:22:58.656246 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:22:59.599634 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.599606 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:22:59.609910 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.609894 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-kube-api-access-m8mc8\" (UniqueName: \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-kube-api-access-m8mc8\") pod \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " Nov 03 01:22:59.609989 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.609933 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istiod-ca-cert\") pod \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " Nov 03 01:22:59.609989 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.609953 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-credential-socket\") pod \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " Nov 03 01:22:59.610108 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610036 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-token\" (UniqueName: \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-token\") pod \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " Nov 03 01:22:59.610108 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610078 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-data\") pod \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " Nov 03 01:22:59.610217 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610118 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-certs\") pod \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " Nov 03 01:22:59.610217 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610141 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-credential-socket" pod "4e77ba6c-2a0e-4224-a790-ade61e6edf66" (UID: "4e77ba6c-2a0e-4224-a790-ade61e6edf66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:59.610217 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610159 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-envoy\") pod \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " Nov 03 01:22:59.610377 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610226 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-podinfo\") pod \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " Nov 03 01:22:59.610377 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610297 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-socket\") pod \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\" (UID: \"4e77ba6c-2a0e-4224-a790-ade61e6edf66\") " Nov 03 01:22:59.610377 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610334 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-data" pod "4e77ba6c-2a0e-4224-a790-ade61e6edf66" (UID: "4e77ba6c-2a0e-4224-a790-ade61e6edf66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:59.610528 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610367 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-certs" pod "4e77ba6c-2a0e-4224-a790-ade61e6edf66" (UID: "4e77ba6c-2a0e-4224-a790-ade61e6edf66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:59.610528 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610372 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istiod-ca-cert" pod "4e77ba6c-2a0e-4224-a790-ade61e6edf66" (UID: "4e77ba6c-2a0e-4224-a790-ade61e6edf66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:59.610627 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610589 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-socket" pod "4e77ba6c-2a0e-4224-a790-ade61e6edf66" (UID: "4e77ba6c-2a0e-4224-a790-ade61e6edf66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:59.610683 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610664 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:59.610750 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610688 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:59.610750 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610704 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:59.610750 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.610717 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:59.612202 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.612181 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-kube-api-access-m8mc8" pod "4e77ba6c-2a0e-4224-a790-ade61e6edf66" (UID: "4e77ba6c-2a0e-4224-a790-ade61e6edf66"). InnerVolumeSpecName "kube-api-access-m8mc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:59.612371 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.612349 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-envoy" pod "4e77ba6c-2a0e-4224-a790-ade61e6edf66" (UID: "4e77ba6c-2a0e-4224-a790-ade61e6edf66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:59.612463 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.612385 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-token" pod "4e77ba6c-2a0e-4224-a790-ade61e6edf66" (UID: "4e77ba6c-2a0e-4224-a790-ade61e6edf66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:59.612463 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.612394 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-podinfo" pod "4e77ba6c-2a0e-4224-a790-ade61e6edf66" (UID: "4e77ba6c-2a0e-4224-a790-ade61e6edf66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:22:59.712066 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.712043 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:59.712066 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.712064 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8mc8\" (UniqueName: \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-kube-api-access-m8mc8\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:59.712423 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.712076 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:59.712423 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.712085 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:59.712423 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:22:59.712093 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e77ba6c-2a0e-4224-a790-ade61e6edf66-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:00.428961 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:00.428926 4304 generic.go:356] "Generic (PLEG): container finished" podID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerID="30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde" exitCode=0 Nov 03 01:23:00.429129 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:00.428973 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" Nov 03 01:23:00.429129 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:00.429011 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" event={"ID":"4e77ba6c-2a0e-4224-a790-ade61e6edf66","Type":"ContainerDied","Data":"30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde"} Nov 03 01:23:00.429129 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:00.429065 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j" event={"ID":"4e77ba6c-2a0e-4224-a790-ade61e6edf66","Type":"ContainerDied","Data":"fc221a61ddfa171579733d47455f602b2610d911cd8b5dd69488cd02aa3a2b5e"} Nov 03 01:23:00.429129 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:00.429095 4304 scope.go:122] "RemoveContainer" containerID="30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde" Nov 03 01:23:00.438118 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:00.438100 4304 scope.go:122] "RemoveContainer" containerID="30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde" Nov 03 01:23:00.438394 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:23:00.438372 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde\": container with ID starting with 30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde not found: ID does not exist" containerID="30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde" Nov 03 01:23:00.438458 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:00.438414 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde"} err="failed to get container status \"30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde\": rpc error: code = NotFound desc = could not find container \"30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde\": container with ID starting with 30ecd94b27ae10464302137b2870c3586aac9015a3b0d2c696a55f2b786addde not found: ID does not exist" Nov 03 01:23:00.640113 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:00.640083 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j"] Nov 03 01:23:00.689236 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:00.689178 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c7z2j"] Nov 03 01:23:00.828097 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:00.828067 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" path="/var/lib/kubelet/pods/4e77ba6c-2a0e-4224-a790-ade61e6edf66/volumes" Nov 03 01:23:08.408893 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.408855 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b"] Nov 03 01:23:08.409278 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.409196 4304 cpu_manager.go:414] "containerMap: removing container" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerName="istio-proxy" Nov 03 01:23:08.409278 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.409206 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerName="istio-proxy" Nov 03 01:23:08.409278 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.409222 4304 cpu_manager.go:414] "containerMap: removing container" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" Nov 03 01:23:08.409278 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.409227 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf51c557-8e65-4a4d-8746-35df22b465e6" containerName="istio-proxy" Nov 03 01:23:08.409278 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.409240 4304 cpu_manager.go:414] "containerMap: removing container" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" Nov 03 01:23:08.409278 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.409245 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" Nov 03 01:23:08.409483 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.409299 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e77ba6c-2a0e-4224-a790-ade61e6edf66" containerName="istio-proxy" Nov 03 01:23:08.409483 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.409310 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab4e1dd-5ed3-4ea0-b315-4ca2001f73a9" containerName="istio-proxy" Nov 03 01:23:08.413753 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.413736 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:08.490139 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.490111 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-plugins-config-volume\") pod \"payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b\" (UID: \"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26\") " pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:08.490311 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.490256 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcsmv\" (UniqueName: \"kubernetes.io/projected/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-kube-api-access-fcsmv\") pod \"payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b\" (UID: \"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26\") " pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:08.514965 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.514942 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-337498b9\"" Nov 03 01:23:08.515116 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.514966 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-337498b9-dockercfg-xd47q\"" Nov 03 01:23:08.516749 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.516728 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b"] Nov 03 01:23:08.591342 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.591313 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-plugins-config-volume\") pod \"payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b\" (UID: \"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26\") " pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:08.591460 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.591396 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcsmv\" (UniqueName: \"kubernetes.io/projected/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-kube-api-access-fcsmv\") pod \"payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b\" (UID: \"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26\") " pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:08.592001 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.591982 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-plugins-config-volume\") pod \"payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b\" (UID: \"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26\") " pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:08.647725 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.647697 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcsmv\" (UniqueName: \"kubernetes.io/projected/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-kube-api-access-fcsmv\") pod \"payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b\" (UID: \"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26\") " pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:08.726552 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:08.726486 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:09.095290 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:09.095268 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b"] Nov 03 01:23:09.097873 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:23:09.097846 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d1e77a_0ea4_4309_bf4e_05dd0f9b5c26.slice/crio-7b65cdadd79abb26c441331fcb4466feaf284056003e504a041326424a0b8359 WatchSource:0}: Error finding container 7b65cdadd79abb26c441331fcb4466feaf284056003e504a041326424a0b8359: Status 404 returned error can't find the container with id 7b65cdadd79abb26c441331fcb4466feaf284056003e504a041326424a0b8359 Nov 03 01:23:09.473391 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:09.473356 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" event={"ID":"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26","Type":"ContainerStarted","Data":"cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c"} Nov 03 01:23:09.473840 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:09.473409 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" event={"ID":"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26","Type":"ContainerStarted","Data":"7b65cdadd79abb26c441331fcb4466feaf284056003e504a041326424a0b8359"} Nov 03 01:23:09.473840 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:09.473501 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:09.585241 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:09.585197 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" podStartSLOduration=1.585184307 podStartE2EDuration="1.585184307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:09.584580793 +0000 UTC m=+4672.868138466" watchObservedRunningTime="2026-11-03 01:23:09.585184307 +0000 UTC m=+4672.868741979" Nov 03 01:23:10.550375 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.550345 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr"] Nov 03 01:23:10.553644 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.553628 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:10.605069 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.605046 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-337498b9-serving-cert\"" Nov 03 01:23:10.610544 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.610515 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhfwl\" (UniqueName: \"kubernetes.io/projected/b9a0944f-c771-434a-bd43-b604ec81d33b-kube-api-access-dhfwl\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:10.610669 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.610570 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-metrics-certs\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:10.610796 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.610751 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a0944f-c771-434a-bd43-b604ec81d33b-openshift-service-ca\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:10.610932 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.610803 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-maas-api-tls\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:10.656457 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.656429 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr"] Nov 03 01:23:10.712285 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.712258 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a0944f-c771-434a-bd43-b604ec81d33b-openshift-service-ca\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:10.712463 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.712299 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-maas-api-tls\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:10.712463 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.712354 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhfwl\" (UniqueName: \"kubernetes.io/projected/b9a0944f-c771-434a-bd43-b604ec81d33b-kube-api-access-dhfwl\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:10.712463 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.712399 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-metrics-certs\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:10.712592 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:23:10.712497 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-serving-cert: secret "maas-api-e2e-ait-337498b9-serving-cert" not found Nov 03 01:23:10.712592 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:23:10.712562 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-maas-api-tls podName:b9a0944f-c771-434a-bd43-b604ec81d33b nodeName:}" failed. No retries permitted until 2026-11-03 01:23:11.212543829 +0000 UTC m=+4674.496101499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-maas-api-tls") pod "maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" (UID: "b9a0944f-c771-434a-bd43-b604ec81d33b") : secret "maas-api-e2e-ait-337498b9-serving-cert" not found Nov 03 01:23:10.712953 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.712929 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a0944f-c771-434a-bd43-b604ec81d33b-openshift-service-ca\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:10.715144 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.715122 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-metrics-certs\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:10.769024 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:10.768991 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhfwl\" (UniqueName: \"kubernetes.io/projected/b9a0944f-c771-434a-bd43-b604ec81d33b-kube-api-access-dhfwl\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:11.217200 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:11.217160 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-maas-api-tls\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:11.219545 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:11.219517 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-maas-api-tls\") pod \"maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:11.465408 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:11.465376 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:11.835250 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:11.835228 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr"] Nov 03 01:23:11.837627 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:23:11.837603 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a0944f_c771_434a_bd43_b604ec81d33b.slice/crio-9683033208d3ac00bf6f6ffbbfa46f14aa006f1eb516eb188189171d108c7026 WatchSource:0}: Error finding container 9683033208d3ac00bf6f6ffbbfa46f14aa006f1eb516eb188189171d108c7026: Status 404 returned error can't find the container with id 9683033208d3ac00bf6f6ffbbfa46f14aa006f1eb516eb188189171d108c7026 Nov 03 01:23:11.842127 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:11.842104 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:23:12.489083 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:12.489051 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" event={"ID":"b9a0944f-c771-434a-bd43-b604ec81d33b","Type":"ContainerStarted","Data":"9683033208d3ac00bf6f6ffbbfa46f14aa006f1eb516eb188189171d108c7026"} Nov 03 01:23:13.495859 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:13.495818 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" event={"ID":"b9a0944f-c771-434a-bd43-b604ec81d33b","Type":"ContainerStarted","Data":"51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535"} Nov 03 01:23:13.496248 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:13.495889 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:13.611417 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:13.611363 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" podStartSLOduration=2.7766391649999997 podStartE2EDuration="3.611349754s" totalImagesPullingTime="834.710589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:13.610824414 +0000 UTC m=+4676.894382130" watchObservedRunningTime="2026-11-03 01:23:13.611349754 +0000 UTC m=+4676.894907420" Nov 03 01:23:19.506259 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:19.506233 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:20.481462 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:20.481434 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:28.656055 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:23:28.656017 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:23:42.610131 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:42.610093 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b"] Nov 03 01:23:47.610594 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:47.610543 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" podUID="07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26" containerName="payload-processing" containerID="cri-o://cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c" gracePeriod=25 Nov 03 01:23:47.853232 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:47.853209 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:47.921941 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:47.921919 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-plugins-config-volume\") pod \"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26\" (UID: \"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26\") " Nov 03 01:23:47.922075 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:47.921986 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-kube-api-access-fcsmv\" (UniqueName: \"kubernetes.io/projected/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-kube-api-access-fcsmv\") pod \"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26\" (UID: \"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26\") " Nov 03 01:23:47.922320 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:47.922297 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-plugins-config-volume" pod "07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26" (UID: "07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:23:47.922397 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:47.922387 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:47.924098 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:47.924075 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-kube-api-access-fcsmv" pod "07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26" (UID: "07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26"). InnerVolumeSpecName "kube-api-access-fcsmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:23:48.023669 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.023642 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcsmv\" (UniqueName: \"kubernetes.io/projected/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26-kube-api-access-fcsmv\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:48.664512 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.664472 4304 generic.go:356] "Generic (PLEG): container finished" podID="07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26" containerID="cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c" exitCode=0 Nov 03 01:23:48.664927 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.664523 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" Nov 03 01:23:48.664927 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.664531 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" event={"ID":"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26","Type":"ContainerDied","Data":"cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c"} Nov 03 01:23:48.664927 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.664583 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b" event={"ID":"07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26","Type":"ContainerDied","Data":"7b65cdadd79abb26c441331fcb4466feaf284056003e504a041326424a0b8359"} Nov 03 01:23:48.664927 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.664613 4304 scope.go:122] "RemoveContainer" containerID="cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c" Nov 03 01:23:48.673229 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.673210 4304 scope.go:122] "RemoveContainer" containerID="cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c" Nov 03 01:23:48.673479 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:23:48.673460 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c\": container with ID starting with cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c not found: ID does not exist" containerID="cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c" Nov 03 01:23:48.673531 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.673489 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c"} err="failed to get container status \"cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c\": rpc error: code = NotFound desc = could not find container \"cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c\": container with ID starting with cd2413edb848ca4a12ee2d521d326338f3edcb186eddc3d81294364f977b212c not found: ID does not exist" Nov 03 01:23:48.685406 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.685381 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr"] Nov 03 01:23:48.685574 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.685557 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" podUID="b9a0944f-c771-434a-bd43-b604ec81d33b" containerName="maas-api" containerID="cri-o://51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535" gracePeriod=30 Nov 03 01:23:48.842887 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.842851 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b"] Nov 03 01:23:48.859596 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.859574 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-337498b9-6b7b884fd5-8ht5b"] Nov 03 01:23:48.933292 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:48.933266 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:49.033001 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.032975 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a0944f-c771-434a-bd43-b604ec81d33b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a0944f-c771-434a-bd43-b604ec81d33b-openshift-service-ca\") pod \"b9a0944f-c771-434a-bd43-b604ec81d33b\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " Nov 03 01:23:49.033151 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.033043 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a0944f-c771-434a-bd43-b604ec81d33b-kube-api-access-dhfwl\" (UniqueName: \"kubernetes.io/projected/b9a0944f-c771-434a-bd43-b604ec81d33b-kube-api-access-dhfwl\") pod \"b9a0944f-c771-434a-bd43-b604ec81d33b\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " Nov 03 01:23:49.033151 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.033091 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-maas-api-tls\") pod \"b9a0944f-c771-434a-bd43-b604ec81d33b\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " Nov 03 01:23:49.033151 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.033135 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-metrics-certs\") pod \"b9a0944f-c771-434a-bd43-b604ec81d33b\" (UID: \"b9a0944f-c771-434a-bd43-b604ec81d33b\") " Nov 03 01:23:49.033426 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.033400 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a0944f-c771-434a-bd43-b604ec81d33b-openshift-service-ca" pod "b9a0944f-c771-434a-bd43-b604ec81d33b" (UID: "b9a0944f-c771-434a-bd43-b604ec81d33b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:23:49.035285 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.035255 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-metrics-certs" pod "b9a0944f-c771-434a-bd43-b604ec81d33b" (UID: "b9a0944f-c771-434a-bd43-b604ec81d33b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:23:49.035382 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.035290 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-maas-api-tls" pod "b9a0944f-c771-434a-bd43-b604ec81d33b" (UID: "b9a0944f-c771-434a-bd43-b604ec81d33b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:23:49.035382 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.035337 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a0944f-c771-434a-bd43-b604ec81d33b-kube-api-access-dhfwl" pod "b9a0944f-c771-434a-bd43-b604ec81d33b" (UID: "b9a0944f-c771-434a-bd43-b604ec81d33b"). InnerVolumeSpecName "kube-api-access-dhfwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:23:49.134495 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.134462 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:49.134495 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.134490 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a0944f-c771-434a-bd43-b604ec81d33b-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:49.134690 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.134502 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhfwl\" (UniqueName: \"kubernetes.io/projected/b9a0944f-c771-434a-bd43-b604ec81d33b-kube-api-access-dhfwl\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:49.134690 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.134515 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a0944f-c771-434a-bd43-b604ec81d33b-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:49.670666 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.670631 4304 generic.go:356] "Generic (PLEG): container finished" podID="b9a0944f-c771-434a-bd43-b604ec81d33b" containerID="51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535" exitCode=0 Nov 03 01:23:49.671131 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.670677 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" Nov 03 01:23:49.671131 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.670720 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" event={"ID":"b9a0944f-c771-434a-bd43-b604ec81d33b","Type":"ContainerDied","Data":"51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535"} Nov 03 01:23:49.671131 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.670767 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr" event={"ID":"b9a0944f-c771-434a-bd43-b604ec81d33b","Type":"ContainerDied","Data":"9683033208d3ac00bf6f6ffbbfa46f14aa006f1eb516eb188189171d108c7026"} Nov 03 01:23:49.671131 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.670811 4304 scope.go:122] "RemoveContainer" containerID="51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535" Nov 03 01:23:49.681078 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.681051 4304 scope.go:122] "RemoveContainer" containerID="51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535" Nov 03 01:23:49.681317 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:23:49.681296 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535\": container with ID starting with 51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535 not found: ID does not exist" containerID="51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535" Nov 03 01:23:49.681370 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.681324 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535"} err="failed to get container status \"51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535\": rpc error: code = NotFound desc = could not find container \"51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535\": container with ID starting with 51ccb1e25f37ddb0e0d9f2b5299a88f5510133a5345dfbc487eb7a7a00891535 not found: ID does not exist" Nov 03 01:23:49.832950 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.832921 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr"] Nov 03 01:23:49.836708 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:49.836685 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-337498b9-5fcdfd8b6b-mvqtr"] Nov 03 01:23:50.827378 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:50.827346 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26" path="/var/lib/kubelet/pods/07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26/volumes" Nov 03 01:23:50.827750 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:23:50.827726 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a0944f-c771-434a-bd43-b604ec81d33b" path="/var/lib/kubelet/pods/b9a0944f-c771-434a-bd43-b604ec81d33b/volumes" Nov 03 01:23:58.656579 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:23:58.656533 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:24:25.053342 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.053312 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x"] Nov 03 01:24:25.053741 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.053649 4304 cpu_manager.go:414] "containerMap: removing container" podUID="07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26" containerName="payload-processing" Nov 03 01:24:25.053741 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.053660 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26" containerName="payload-processing" Nov 03 01:24:25.053741 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.053686 4304 cpu_manager.go:414] "containerMap: removing container" podUID="b9a0944f-c771-434a-bd43-b604ec81d33b" containerName="maas-api" Nov 03 01:24:25.053741 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.053691 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a0944f-c771-434a-bd43-b604ec81d33b" containerName="maas-api" Nov 03 01:24:25.053900 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.053745 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a0944f-c771-434a-bd43-b604ec81d33b" containerName="maas-api" Nov 03 01:24:25.053900 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.053755 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d1e77a-0ea4-4309-bf4e-05dd0f9b5c26" containerName="payload-processing" Nov 03 01:24:25.058236 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.058204 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.169890 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.169856 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x"] Nov 03 01:24:25.238999 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.238974 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.239149 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.239002 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.239149 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.239020 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.239149 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.239041 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crtlc\" (UniqueName: \"kubernetes.io/projected/5bb4aa28-be0e-4705-9133-23afa41c0701-kube-api-access-crtlc\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.239269 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.239179 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb4aa28-be0e-4705-9133-23afa41c0701-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.239269 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.239219 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.239269 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.239239 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.239371 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.239319 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.239371 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.239351 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.340634 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.340561 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb4aa28-be0e-4705-9133-23afa41c0701-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.340634 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.340610 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.340634 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.340631 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.340915 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.340664 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.340915 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.340698 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.340915 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.340741 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.340915 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.340761 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.340915 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.340811 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.340915 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.340843 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crtlc\" (UniqueName: \"kubernetes.io/projected/5bb4aa28-be0e-4705-9133-23afa41c0701-kube-api-access-crtlc\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.341200 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.341046 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.341200 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.341086 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.341285 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.341205 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.341285 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.341219 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.341362 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.341302 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb4aa28-be0e-4705-9133-23afa41c0701-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.343141 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.343123 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.343382 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.343366 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.396904 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.396876 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crtlc\" (UniqueName: \"kubernetes.io/projected/5bb4aa28-be0e-4705-9133-23afa41c0701-kube-api-access-crtlc\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.396990 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.396916 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb4aa28-be0e-4705-9133-23afa41c0701-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nk66x\" (UID: \"5bb4aa28-be0e-4705-9133-23afa41c0701\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:25.669969 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:25.669941 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:26.093825 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:26.093799 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x"] Nov 03 01:24:26.096087 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:24:26.096061 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb4aa28_be0e_4705_9133_23afa41c0701.slice/crio-d42b8e6a464a7235f3f1ed71895df63efd5644167619d6c3d9d353b081bc14e8 WatchSource:0}: Error finding container d42b8e6a464a7235f3f1ed71895df63efd5644167619d6c3d9d353b081bc14e8: Status 404 returned error can't find the container with id d42b8e6a464a7235f3f1ed71895df63efd5644167619d6c3d9d353b081bc14e8 Nov 03 01:24:26.098054 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:26.098020 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:24:26.098160 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:26.098099 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:24:26.098160 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:26.098139 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:24:26.852030 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:26.851986 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" event={"ID":"5bb4aa28-be0e-4705-9133-23afa41c0701","Type":"ContainerStarted","Data":"dd686368ef589ec73ba0382262c859eb1b6e5880bd87664e784f2662229fd01e"} Nov 03 01:24:26.852216 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:26.852046 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" event={"ID":"5bb4aa28-be0e-4705-9133-23afa41c0701","Type":"ContainerStarted","Data":"d42b8e6a464a7235f3f1ed71895df63efd5644167619d6c3d9d353b081bc14e8"} Nov 03 01:24:27.017004 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:27.016952 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" podStartSLOduration=2.016936636 podStartE2EDuration="2.016936636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:27.015671474 +0000 UTC m=+4750.299229145" watchObservedRunningTime="2026-11-03 01:24:27.016936636 +0000 UTC m=+4750.300494309" Nov 03 01:24:27.671089 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:27.671052 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:27.672425 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:27.672403 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:24:27.672554 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:27.672447 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" podUID="5bb4aa28-be0e-4705-9133-23afa41c0701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 01:24:28.670348 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:28.670314 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:24:28.670547 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:28.670378 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" podUID="5bb4aa28-be0e-4705-9133-23afa41c0701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 01:24:28.670547 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:24:28.670488 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:24:29.670646 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:29.670615 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:24:29.671063 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:29.670665 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" podUID="5bb4aa28-be0e-4705-9133-23afa41c0701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 01:24:30.671233 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:30.671198 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:24:30.671639 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:30.671249 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" podUID="5bb4aa28-be0e-4705-9133-23afa41c0701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 01:24:31.670229 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:31.670195 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:24:31.670410 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:31.670252 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" podUID="5bb4aa28-be0e-4705-9133-23afa41c0701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 01:24:32.670294 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:32.670257 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 01:24:32.670683 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:32.670322 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" podUID="5bb4aa28-be0e-4705-9133-23afa41c0701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 01:24:34.671033 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:34.670995 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:24:34.671433 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:34.671060 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" podUID="5bb4aa28-be0e-4705-9133-23afa41c0701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:35.711938 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:35.711904 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:24:35.712427 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:35.711970 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" podUID="5bb4aa28-be0e-4705-9133-23afa41c0701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:36.260181 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:36.260151 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:36.260454 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:36.260435 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:36.461416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:24:36.461386 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" Nov 03 01:24:58.653918 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:24:58.653881 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:25:25.500016 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.499987 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2"] Nov 03 01:25:25.503426 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.503409 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.549862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.549838 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z"] Nov 03 01:25:25.553055 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.553039 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.560726 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.560703 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.560853 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.560801 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.560929 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.560874 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.560929 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.560908 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.561019 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.560971 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.561019 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.560992 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmsb4\" (UniqueName: \"kubernetes.io/projected/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-kube-api-access-kmsb4\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.561019 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561016 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.561181 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561037 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.561181 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561096 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.561181 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561162 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.561332 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561197 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.561332 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561225 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.561332 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561285 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.561332 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561321 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.561483 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561353 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.561483 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561429 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvq2w\" (UniqueName: \"kubernetes.io/projected/ab6138a0-6598-4db3-a290-cbc0d3f67f05-kube-api-access-lvq2w\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.561483 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561469 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.561743 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.561491 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.662592 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.662556 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.662592 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.662598 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmsb4\" (UniqueName: \"kubernetes.io/projected/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-kube-api-access-kmsb4\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.662791 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.662635 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.662848 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.662817 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.662896 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.662852 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.662945 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.662896 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.662945 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.662930 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.663068 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663042 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.663126 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663075 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.663178 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663133 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.663178 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663170 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.663287 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663208 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.663287 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663212 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.663287 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663240 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvq2w\" (UniqueName: \"kubernetes.io/projected/ab6138a0-6598-4db3-a290-cbc0d3f67f05-kube-api-access-lvq2w\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.663287 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663269 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.663511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663302 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.663511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663341 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.663511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663375 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.663511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663389 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.663511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663410 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.663511 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663436 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.664107 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663660 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.664107 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663768 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.664107 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.663409 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.664107 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.664019 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.664347 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.664321 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.664497 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.664468 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.664703 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.664676 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.665791 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.665727 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.666123 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.666102 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.666248 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.666226 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.666471 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.666454 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.695991 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.695966 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z"] Nov 03 01:25:25.719089 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.719068 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2"] Nov 03 01:25:25.723851 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.723821 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmsb4\" (UniqueName: \"kubernetes.io/projected/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-kube-api-access-kmsb4\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.724383 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.724360 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-88r4z\" (UID: \"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:25.725040 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.725009 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab6138a0-6598-4db3-a290-cbc0d3f67f05-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.725314 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.725299 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvq2w\" (UniqueName: \"kubernetes.io/projected/ab6138a0-6598-4db3-a290-cbc0d3f67f05-kube-api-access-lvq2w\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2\" (UID: \"ab6138a0-6598-4db3-a290-cbc0d3f67f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.816100 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.816038 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:25.865819 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:25.864702 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:26.236725 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:26.236691 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2"] Nov 03 01:25:26.278691 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:25:26.278664 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6138a0_6598_4db3_a290_cbc0d3f67f05.slice/crio-a8dbd5d8e0fa8f9a77ca396ecf2d66ba2411e224c5c725f1cd80fc4c90ca3a0d WatchSource:0}: Error finding container a8dbd5d8e0fa8f9a77ca396ecf2d66ba2411e224c5c725f1cd80fc4c90ca3a0d: Status 404 returned error can't find the container with id a8dbd5d8e0fa8f9a77ca396ecf2d66ba2411e224c5c725f1cd80fc4c90ca3a0d Nov 03 01:25:26.280989 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:26.280936 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:25:26.281082 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:26.281043 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:25:26.281124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:26.281090 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:25:26.289075 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:26.289053 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z"] Nov 03 01:25:26.290762 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:25:26.290734 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecd91cb_b063_43aa_9a47_b7dc57a6a2c6.slice/crio-efef592ababe26b95a85df81d9101fde8765a6e32f20f88902624c87d69ea99e WatchSource:0}: Error finding container efef592ababe26b95a85df81d9101fde8765a6e32f20f88902624c87d69ea99e: Status 404 returned error can't find the container with id efef592ababe26b95a85df81d9101fde8765a6e32f20f88902624c87d69ea99e Nov 03 01:25:26.293142 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:26.293115 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:25:26.293236 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:26.293168 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:25:26.293236 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:26.293194 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:25:27.156801 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:27.148728 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" event={"ID":"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6","Type":"ContainerStarted","Data":"1794894a986793bcbc37a61584844cdf0c5a5e148f0d4beb3e51abb2462fa5b5"} Nov 03 01:25:27.156801 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:27.148826 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" event={"ID":"4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6","Type":"ContainerStarted","Data":"efef592ababe26b95a85df81d9101fde8765a6e32f20f88902624c87d69ea99e"} Nov 03 01:25:27.156801 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:27.156556 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" event={"ID":"ab6138a0-6598-4db3-a290-cbc0d3f67f05","Type":"ContainerStarted","Data":"6d7e865c9ace8a34afec572131e8206d162dded49d93f491bc3eb012bf0f43e6"} Nov 03 01:25:27.156801 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:27.156635 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" event={"ID":"ab6138a0-6598-4db3-a290-cbc0d3f67f05","Type":"ContainerStarted","Data":"a8dbd5d8e0fa8f9a77ca396ecf2d66ba2411e224c5c725f1cd80fc4c90ca3a0d"} Nov 03 01:25:27.377853 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:27.377792 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" podStartSLOduration=2.377754348 podStartE2EDuration="2.377754348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:27.320918548 +0000 UTC m=+4810.604476223" watchObservedRunningTime="2026-11-03 01:25:27.377754348 +0000 UTC m=+4810.661312037" Nov 03 01:25:27.823911 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:27.823877 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:27.824264 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:27.824241 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 01:25:27.824429 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:27.824405 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" podUID="ab6138a0-6598-4db3-a290-cbc0d3f67f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 01:25:27.870595 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:27.870369 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:27.870595 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:27.870483 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 01:25:27.870595 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:27.870531 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" podUID="4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 01:25:28.796051 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:25:28.796009 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:25:28.817439 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:28.817408 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 01:25:28.817710 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:28.817682 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" podUID="ab6138a0-6598-4db3-a290-cbc0d3f67f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 01:25:28.869128 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:28.869094 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 01:25:28.869422 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:28.869362 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" podUID="4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 01:25:29.816457 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:29.816372 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 01:25:29.816457 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:29.816421 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" podUID="ab6138a0-6598-4db3-a290-cbc0d3f67f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 01:25:29.871668 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:29.871632 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 01:25:29.871954 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:29.871924 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" podUID="4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 01:25:30.817179 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:30.817142 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 01:25:30.817577 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:30.817206 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" podUID="ab6138a0-6598-4db3-a290-cbc0d3f67f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 01:25:30.874212 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:30.874174 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 01:25:30.874409 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:30.874244 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" podUID="4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 01:25:31.817106 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:31.817068 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 01:25:31.817276 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:31.817142 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" podUID="ab6138a0-6598-4db3-a290-cbc0d3f67f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 01:25:31.870232 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:31.870193 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 01:25:31.870507 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:31.870475 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" podUID="4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 01:25:32.817488 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:32.817450 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 01:25:32.817912 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:32.817507 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" podUID="ab6138a0-6598-4db3-a290-cbc0d3f67f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 01:25:32.866171 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:32.866137 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 01:25:32.866371 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:32.866197 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" podUID="4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 01:25:33.816507 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:33.816470 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 01:25:33.816680 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:33.816536 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" podUID="ab6138a0-6598-4db3-a290-cbc0d3f67f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 01:25:34.633557 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:34.633521 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:34.634630 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:34.634603 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:34.635247 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:34.635224 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" Nov 03 01:25:34.799687 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:34.799634 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" podStartSLOduration=9.799617399 podStartE2EDuration="9.799617399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:27.450281761 +0000 UTC m=+4810.733839435" watchObservedRunningTime="2026-11-03 01:25:34.799617399 +0000 UTC m=+4818.083175073" Nov 03 01:25:35.301791 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:35.301744 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:35.302273 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:35.302249 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:35.303024 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:25:35.303001 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" Nov 03 01:25:58.653406 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:25:58.653368 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:26:28.655334 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:26:28.655297 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:26:58.654821 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:26:58.654769 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:27:28.654253 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:27:28.654218 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:27:57.101830 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.101798 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn"] Nov 03 01:27:57.105341 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.105320 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.160569 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.160535 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-6c9df1-openshift-default-dockercfg-4q2fd\"" Nov 03 01:27:57.200861 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.200830 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-credential-socket\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.200861 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.200866 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-certs\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.201103 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.200891 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-data\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.201103 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.200983 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-socket\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.201103 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.201014 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-istio-token\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.201276 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.201105 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d962085-93d5-4d03-99c9-a3931510c10d-istio-podinfo\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.201276 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.201148 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljx7\" (UniqueName: \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-kube-api-access-8ljx7\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.201276 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.201164 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-envoy\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.201276 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.201232 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d962085-93d5-4d03-99c9-a3931510c10d-istiod-ca-cert\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.212514 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.212480 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn"] Nov 03 01:27:57.302196 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.302157 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d962085-93d5-4d03-99c9-a3931510c10d-istio-podinfo\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.302403 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.302233 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljx7\" (UniqueName: \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-kube-api-access-8ljx7\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.302403 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.302259 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-envoy\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.302403 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.302312 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d962085-93d5-4d03-99c9-a3931510c10d-istiod-ca-cert\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.302403 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.302357 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-credential-socket\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.302403 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.302382 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-certs\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.302678 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.302418 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-data\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.302678 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.302454 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-socket\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.302678 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.302486 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-istio-token\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.302912 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.302833 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-credential-socket\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.302969 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.302923 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-certs\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.303195 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.303137 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-socket\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.303195 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.303192 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d962085-93d5-4d03-99c9-a3931510c10d-istiod-ca-cert\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.303195 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.303200 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-data\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.304697 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.304675 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-envoy\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.304972 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.304955 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d962085-93d5-4d03-99c9-a3931510c10d-istio-podinfo\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.358247 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.358190 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-istio-token\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.358392 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.358376 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljx7\" (UniqueName: \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-kube-api-access-8ljx7\") pod \"e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.417592 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.417565 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:57.839475 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.839455 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn"] Nov 03 01:27:57.841095 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:27:57.841071 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d962085_93d5_4d03_99c9_a3931510c10d.slice/crio-8c227d86050039b47e194c6e7e07a43126f4cba438cb7c499db6a4ed9ab1d19f WatchSource:0}: Error finding container 8c227d86050039b47e194c6e7e07a43126f4cba438cb7c499db6a4ed9ab1d19f: Status 404 returned error can't find the container with id 8c227d86050039b47e194c6e7e07a43126f4cba438cb7c499db6a4ed9ab1d19f Nov 03 01:27:57.843138 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.843107 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:27:57.843228 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.843169 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:27:57.843228 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:57.843203 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:27:58.015035 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:58.014994 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" event={"ID":"1d962085-93d5-4d03-99c9-a3931510c10d","Type":"ContainerStarted","Data":"51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8"} Nov 03 01:27:58.015233 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:58.015052 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" event={"ID":"1d962085-93d5-4d03-99c9-a3931510c10d","Type":"ContainerStarted","Data":"8c227d86050039b47e194c6e7e07a43126f4cba438cb7c499db6a4ed9ab1d19f"} Nov 03 01:27:58.134593 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:58.134498 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" podStartSLOduration=1.134484709 podStartE2EDuration="1.134484709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:58.131524583 +0000 UTC m=+4961.415082258" watchObservedRunningTime="2026-11-03 01:27:58.134484709 +0000 UTC m=+4961.418042381" Nov 03 01:27:58.424799 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:58.420863 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:58.437135 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:58.437102 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:58.774753 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:27:58.773908 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:27:59.025097 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:59.025018 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:27:59.026917 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:27:59.026895 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:28:05.024872 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.024836 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv"] Nov 03 01:28:05.031030 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.031004 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:28:05.071472 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.071438 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5jg\" (UniqueName: \"kubernetes.io/projected/a4b646e7-1966-4903-9141-668dc348de25-kube-api-access-dx5jg\") pod \"payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv\" (UID: \"a4b646e7-1966-4903-9141-668dc348de25\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:28:05.071642 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.071511 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b646e7-1966-4903-9141-668dc348de25-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv\" (UID: \"a4b646e7-1966-4903-9141-668dc348de25\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:28:05.084959 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.084935 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv"] Nov 03 01:28:05.132204 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.132174 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-6c9df1\"" Nov 03 01:28:05.132318 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.132264 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-29a92afa-dockercfg-vbfjl\"" Nov 03 01:28:05.172461 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.172437 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5jg\" (UniqueName: \"kubernetes.io/projected/a4b646e7-1966-4903-9141-668dc348de25-kube-api-access-dx5jg\") pod \"payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv\" (UID: \"a4b646e7-1966-4903-9141-668dc348de25\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:28:05.172565 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.172493 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b646e7-1966-4903-9141-668dc348de25-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv\" (UID: \"a4b646e7-1966-4903-9141-668dc348de25\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:28:05.173189 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.173171 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b646e7-1966-4903-9141-668dc348de25-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv\" (UID: \"a4b646e7-1966-4903-9141-668dc348de25\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:28:05.204589 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.204569 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c"] Nov 03 01:28:05.207994 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.207980 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:28:05.228466 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.228436 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5jg\" (UniqueName: \"kubernetes.io/projected/a4b646e7-1966-4903-9141-668dc348de25-kube-api-access-dx5jg\") pod \"payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv\" (UID: \"a4b646e7-1966-4903-9141-668dc348de25\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:28:05.265360 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.265336 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c"] Nov 03 01:28:05.273309 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.273282 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mslkt\" (UniqueName: \"kubernetes.io/projected/f6cc93d7-802e-47c6-bcc5-562466d26228-kube-api-access-mslkt\") pod \"payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c\" (UID: \"f6cc93d7-802e-47c6-bcc5-562466d26228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:28:05.273420 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.273325 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc93d7-802e-47c6-bcc5-562466d26228-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c\" (UID: \"f6cc93d7-802e-47c6-bcc5-562466d26228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:28:05.347821 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.347741 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:28:05.373899 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.373876 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslkt\" (UniqueName: \"kubernetes.io/projected/f6cc93d7-802e-47c6-bcc5-562466d26228-kube-api-access-mslkt\") pod \"payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c\" (UID: \"f6cc93d7-802e-47c6-bcc5-562466d26228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:28:05.374022 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.373907 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc93d7-802e-47c6-bcc5-562466d26228-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c\" (UID: \"f6cc93d7-802e-47c6-bcc5-562466d26228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:28:05.374562 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.374542 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc93d7-802e-47c6-bcc5-562466d26228-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c\" (UID: \"f6cc93d7-802e-47c6-bcc5-562466d26228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:28:05.430453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.430423 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslkt\" (UniqueName: \"kubernetes.io/projected/f6cc93d7-802e-47c6-bcc5-562466d26228-kube-api-access-mslkt\") pod \"payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c\" (UID: \"f6cc93d7-802e-47c6-bcc5-562466d26228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:28:05.521140 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.521108 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:28:05.718890 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.718866 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv"] Nov 03 01:28:05.720968 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:28:05.720941 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b646e7_1966_4903_9141_668dc348de25.slice/crio-b42102be450e75c2d4210d1eb71b062b8ed02a0070d9161d192238f5f9f316d8 WatchSource:0}: Error finding container b42102be450e75c2d4210d1eb71b062b8ed02a0070d9161d192238f5f9f316d8: Status 404 returned error can't find the container with id b42102be450e75c2d4210d1eb71b062b8ed02a0070d9161d192238f5f9f316d8 Nov 03 01:28:05.891868 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:05.891843 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c"] Nov 03 01:28:06.071207 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:06.071142 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" event={"ID":"a4b646e7-1966-4903-9141-668dc348de25","Type":"ContainerStarted","Data":"52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468"} Nov 03 01:28:06.071207 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:06.071191 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" event={"ID":"a4b646e7-1966-4903-9141-668dc348de25","Type":"ContainerStarted","Data":"b42102be450e75c2d4210d1eb71b062b8ed02a0070d9161d192238f5f9f316d8"} Nov 03 01:28:06.071646 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:06.071251 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:28:06.072850 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:06.072817 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" event={"ID":"f6cc93d7-802e-47c6-bcc5-562466d26228","Type":"ContainerStarted","Data":"c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab"} Nov 03 01:28:06.072986 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:06.072859 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" event={"ID":"f6cc93d7-802e-47c6-bcc5-562466d26228","Type":"ContainerStarted","Data":"080f08f3c1d11175f89d02ad50ed80f5d45610c637d168b25ec5141082c0bb62"} Nov 03 01:28:06.072986 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:06.072921 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:28:06.185118 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:06.185073 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" podStartSLOduration=2.185061217 podStartE2EDuration="2.185061217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:06.183084158 +0000 UTC m=+4969.466641836" watchObservedRunningTime="2026-11-03 01:28:06.185061217 +0000 UTC m=+4969.468618890" Nov 03 01:28:06.299175 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:06.299112 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" podStartSLOduration=1.299092924 podStartE2EDuration="1.299092924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:06.294839565 +0000 UTC m=+4969.578397259" watchObservedRunningTime="2026-11-03 01:28:06.299092924 +0000 UTC m=+4969.582650597" Nov 03 01:28:07.347671 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.347630 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk"] Nov 03 01:28:07.351276 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.351256 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.391576 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.391553 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.391721 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.391589 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9tfg\" (UniqueName: \"kubernetes.io/projected/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-kube-api-access-f9tfg\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.391763 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.391737 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.391833 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.391821 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.403009 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.402986 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-6c9df1-serving-cert\"" Nov 03 01:28:07.455086 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.455059 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk"] Nov 03 01:28:07.492493 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.492455 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.492643 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.492500 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.492643 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.492520 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9tfg\" (UniqueName: \"kubernetes.io/projected/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-kube-api-access-f9tfg\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.492643 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.492589 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.492843 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:28:07.492728 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-serving-cert: secret "maas-api-e2e-worker-single-main-6c9df1-serving-cert" not found Nov 03 01:28:07.492843 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:28:07.492809 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-maas-api-tls podName:f184bec5-3fbb-4dbd-b1a8-5b0fff79e532 nodeName:}" failed. No retries permitted until 2026-11-03 01:28:07.99278872 +0000 UTC m=+4971.276346388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-maas-api-tls") pod "maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" (UID: "f184bec5-3fbb-4dbd-b1a8-5b0fff79e532") : secret "maas-api-e2e-worker-single-main-6c9df1-serving-cert" not found Nov 03 01:28:07.493168 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.493145 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.495189 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.495165 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.553581 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.553546 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9tfg\" (UniqueName: \"kubernetes.io/projected/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-kube-api-access-f9tfg\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:07.998033 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:07.997999 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:08.000475 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:08.000446 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:08.264256 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:08.264176 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:08.635534 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:08.635508 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk"] Nov 03 01:28:08.637191 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:28:08.637169 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf184bec5_3fbb_4dbd_b1a8_5b0fff79e532.slice/crio-a0f13403aecffae10b1149ecf320c28c98fdd7f4755a7b11ef88dff8e0ebe556 WatchSource:0}: Error finding container a0f13403aecffae10b1149ecf320c28c98fdd7f4755a7b11ef88dff8e0ebe556: Status 404 returned error can't find the container with id a0f13403aecffae10b1149ecf320c28c98fdd7f4755a7b11ef88dff8e0ebe556 Nov 03 01:28:09.089569 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:09.089538 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" event={"ID":"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532","Type":"ContainerStarted","Data":"a0f13403aecffae10b1149ecf320c28c98fdd7f4755a7b11ef88dff8e0ebe556"} Nov 03 01:28:10.095932 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:10.095902 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" event={"ID":"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532","Type":"ContainerStarted","Data":"d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7"} Nov 03 01:28:10.096349 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:10.095963 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:10.210466 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:10.210416 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" podStartSLOduration=2.292282728 podStartE2EDuration="3.210401383s" totalImagesPullingTime="918.118655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:10.20871149 +0000 UTC m=+4973.492269164" watchObservedRunningTime="2026-11-03 01:28:10.210401383 +0000 UTC m=+4973.493959056" Nov 03 01:28:16.106047 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:16.106017 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:28:17.082436 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:17.082407 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:28:17.082626 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:17.082489 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:28:25.199478 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:25.199441 4304 generic.go:356] "Generic (PLEG): container finished" podID="1d962085-93d5-4d03-99c9-a3931510c10d" containerID="51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8" exitCode=0 Nov 03 01:28:25.199929 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:25.199500 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" event={"ID":"1d962085-93d5-4d03-99c9-a3931510c10d","Type":"ContainerDied","Data":"51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8"} Nov 03 01:28:25.199929 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:25.199838 4304 scope.go:122] "RemoveContainer" containerID="51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8" Nov 03 01:28:25.201033 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:25.201004 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:28:25.201116 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:25.201079 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:28:25.201156 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:25.201114 4304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:28:26.205868 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:26.205828 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" event={"ID":"1d962085-93d5-4d03-99c9-a3931510c10d","Type":"ContainerStarted","Data":"f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50"} Nov 03 01:28:26.418713 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:26.418681 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:28:26.419826 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:26.419796 4304 patch_prober.go:29] interesting pod/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 01:28:26.419992 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:26.419860 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 01:28:27.417932 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:27.417896 4304 patch_prober.go:29] interesting pod/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 01:28:27.418434 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:27.417960 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 01:28:28.418080 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:28.418047 4304 patch_prober.go:29] interesting pod/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 01:28:28.418550 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:28.418107 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 01:28:28.670379 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:28:28.670298 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:28:29.036849 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:29.029856 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:28:29.418853 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:29.418819 4304 patch_prober.go:29] interesting pod/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 01:28:29.419206 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:29.418879 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 01:28:30.418344 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:30.418303 4304 patch_prober.go:29] interesting pod/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 01:28:30.418515 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:30.418364 4304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 01:28:31.422672 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:31.422645 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:28:31.423743 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:28:31.423721 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:28:58.654605 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:28:58.654567 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:29:28.654964 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:29:28.654925 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:29:33.168561 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:33.168530 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv"] Nov 03 01:29:33.277311 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:33.277279 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c"] Nov 03 01:29:34.188290 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:34.188257 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk"] Nov 03 01:29:34.188710 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:34.188440 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" podUID="f184bec5-3fbb-4dbd-b1a8-5b0fff79e532" containerName="maas-api" containerID="cri-o://d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7" gracePeriod=30 Nov 03 01:29:35.436389 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.436359 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:29:35.570553 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.570482 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-openshift-service-ca\") pod \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " Nov 03 01:29:35.570553 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.570546 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-metrics-certs\") pod \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " Nov 03 01:29:35.570806 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.570636 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-kube-api-access-f9tfg\" (UniqueName: \"kubernetes.io/projected/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-kube-api-access-f9tfg\") pod \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " Nov 03 01:29:35.570806 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.570660 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-maas-api-tls\") pod \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\" (UID: \"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532\") " Nov 03 01:29:35.570950 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.570885 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-openshift-service-ca" pod "f184bec5-3fbb-4dbd-b1a8-5b0fff79e532" (UID: "f184bec5-3fbb-4dbd-b1a8-5b0fff79e532"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:29:35.572750 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.572711 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-metrics-certs" pod "f184bec5-3fbb-4dbd-b1a8-5b0fff79e532" (UID: "f184bec5-3fbb-4dbd-b1a8-5b0fff79e532"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:29:35.572903 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.572749 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-kube-api-access-f9tfg" pod "f184bec5-3fbb-4dbd-b1a8-5b0fff79e532" (UID: "f184bec5-3fbb-4dbd-b1a8-5b0fff79e532"). InnerVolumeSpecName "kube-api-access-f9tfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:29:35.572903 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.572808 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-maas-api-tls" pod "f184bec5-3fbb-4dbd-b1a8-5b0fff79e532" (UID: "f184bec5-3fbb-4dbd-b1a8-5b0fff79e532"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:29:35.638124 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.638095 4304 generic.go:356] "Generic (PLEG): container finished" podID="f184bec5-3fbb-4dbd-b1a8-5b0fff79e532" containerID="d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7" exitCode=0 Nov 03 01:29:35.638254 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.638131 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" Nov 03 01:29:35.638254 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.638169 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" event={"ID":"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532","Type":"ContainerDied","Data":"d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7"} Nov 03 01:29:35.638254 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.638222 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk" event={"ID":"f184bec5-3fbb-4dbd-b1a8-5b0fff79e532","Type":"ContainerDied","Data":"a0f13403aecffae10b1149ecf320c28c98fdd7f4755a7b11ef88dff8e0ebe556"} Nov 03 01:29:35.638254 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.638245 4304 scope.go:122] "RemoveContainer" containerID="d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7" Nov 03 01:29:35.647765 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.647749 4304 scope.go:122] "RemoveContainer" containerID="d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7" Nov 03 01:29:35.648073 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:29:35.648054 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7\": container with ID starting with d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7 not found: ID does not exist" containerID="d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7" Nov 03 01:29:35.648115 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.648085 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7"} err="failed to get container status \"d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7\": rpc error: code = NotFound desc = could not find container \"d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7\": container with ID starting with d22115b9f37523a253a9fc629a3a4d171f5108b7564a4cbe57d658e27c103bc7 not found: ID does not exist" Nov 03 01:29:35.671660 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.671639 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:35.671752 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.671662 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:35.671752 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.671677 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9tfg\" (UniqueName: \"kubernetes.io/projected/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-kube-api-access-f9tfg\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:35.671752 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.671691 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:35.799533 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.799508 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk"] Nov 03 01:29:35.802697 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:35.802678 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c9df1-7d4cd47bf5-whgbk"] Nov 03 01:29:36.828573 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:36.828531 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f184bec5-3fbb-4dbd-b1a8-5b0fff79e532" path="/var/lib/kubelet/pods/f184bec5-3fbb-4dbd-b1a8-5b0fff79e532/volumes" Nov 03 01:29:38.169550 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:38.169509 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" podUID="a4b646e7-1966-4903-9141-668dc348de25" containerName="payload-processing" containerID="cri-o://52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468" gracePeriod=25 Nov 03 01:29:38.277537 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:38.277498 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" podUID="f6cc93d7-802e-47c6-bcc5-562466d26228" containerName="payload-pre-processing" containerID="cri-o://c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab" gracePeriod=25 Nov 03 01:29:47.081302 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:47.081266 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" podUID="f6cc93d7-802e-47c6-bcc5-562466d26228" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:29:47.081762 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:47.081308 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" podUID="a4b646e7-1966-4903-9141-668dc348de25" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:29:57.080498 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:57.080450 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" podUID="f6cc93d7-802e-47c6-bcc5-562466d26228" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:29:57.080498 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:57.080482 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" podUID="a4b646e7-1966-4903-9141-668dc348de25" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:29:58.423230 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.423203 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:29:58.522113 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.522088 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:29:58.585581 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.585552 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b646e7-1966-4903-9141-668dc348de25-kube-api-access-dx5jg\" (UniqueName: \"kubernetes.io/projected/a4b646e7-1966-4903-9141-668dc348de25-kube-api-access-dx5jg\") pod \"a4b646e7-1966-4903-9141-668dc348de25\" (UID: \"a4b646e7-1966-4903-9141-668dc348de25\") " Nov 03 01:29:58.585720 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.585664 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b646e7-1966-4903-9141-668dc348de25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b646e7-1966-4903-9141-668dc348de25-plugins-config-volume\") pod \"a4b646e7-1966-4903-9141-668dc348de25\" (UID: \"a4b646e7-1966-4903-9141-668dc348de25\") " Nov 03 01:29:58.586065 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.586045 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b646e7-1966-4903-9141-668dc348de25-plugins-config-volume" pod "a4b646e7-1966-4903-9141-668dc348de25" (UID: "a4b646e7-1966-4903-9141-668dc348de25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:29:58.587726 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.587708 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b646e7-1966-4903-9141-668dc348de25-kube-api-access-dx5jg" pod "a4b646e7-1966-4903-9141-668dc348de25" (UID: "a4b646e7-1966-4903-9141-668dc348de25"). InnerVolumeSpecName "kube-api-access-dx5jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:29:58.657688 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:29:58.657656 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:29:58.687043 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.687019 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6cc93d7-802e-47c6-bcc5-562466d26228-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc93d7-802e-47c6-bcc5-562466d26228-plugins-config-volume\") pod \"f6cc93d7-802e-47c6-bcc5-562466d26228\" (UID: \"f6cc93d7-802e-47c6-bcc5-562466d26228\") " Nov 03 01:29:58.687165 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.687054 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6cc93d7-802e-47c6-bcc5-562466d26228-kube-api-access-mslkt\" (UniqueName: \"kubernetes.io/projected/f6cc93d7-802e-47c6-bcc5-562466d26228-kube-api-access-mslkt\") pod \"f6cc93d7-802e-47c6-bcc5-562466d26228\" (UID: \"f6cc93d7-802e-47c6-bcc5-562466d26228\") " Nov 03 01:29:58.687373 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.687356 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b646e7-1966-4903-9141-668dc348de25-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:58.687416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.687381 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx5jg\" (UniqueName: \"kubernetes.io/projected/a4b646e7-1966-4903-9141-668dc348de25-kube-api-access-dx5jg\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:58.687453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.687416 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6cc93d7-802e-47c6-bcc5-562466d26228-plugins-config-volume" pod "f6cc93d7-802e-47c6-bcc5-562466d26228" (UID: "f6cc93d7-802e-47c6-bcc5-562466d26228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:29:58.689166 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.689145 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cc93d7-802e-47c6-bcc5-562466d26228-kube-api-access-mslkt" pod "f6cc93d7-802e-47c6-bcc5-562466d26228" (UID: "f6cc93d7-802e-47c6-bcc5-562466d26228"). InnerVolumeSpecName "kube-api-access-mslkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:29:58.775311 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.775236 4304 generic.go:356] "Generic (PLEG): container finished" podID="a4b646e7-1966-4903-9141-668dc348de25" containerID="52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468" exitCode=0 Nov 03 01:29:58.775311 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.775266 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" event={"ID":"a4b646e7-1966-4903-9141-668dc348de25","Type":"ContainerDied","Data":"52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468"} Nov 03 01:29:58.775311 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.775304 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" Nov 03 01:29:58.775552 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.775324 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv" event={"ID":"a4b646e7-1966-4903-9141-668dc348de25","Type":"ContainerDied","Data":"b42102be450e75c2d4210d1eb71b062b8ed02a0070d9161d192238f5f9f316d8"} Nov 03 01:29:58.775552 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.775343 4304 scope.go:122] "RemoveContainer" containerID="52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468" Nov 03 01:29:58.776673 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.776638 4304 generic.go:356] "Generic (PLEG): container finished" podID="f6cc93d7-802e-47c6-bcc5-562466d26228" containerID="c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab" exitCode=0 Nov 03 01:29:58.776805 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.776682 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" Nov 03 01:29:58.776805 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.776693 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" event={"ID":"f6cc93d7-802e-47c6-bcc5-562466d26228","Type":"ContainerDied","Data":"c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab"} Nov 03 01:29:58.776805 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.776744 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c" event={"ID":"f6cc93d7-802e-47c6-bcc5-562466d26228","Type":"ContainerDied","Data":"080f08f3c1d11175f89d02ad50ed80f5d45610c637d168b25ec5141082c0bb62"} Nov 03 01:29:58.785694 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.785665 4304 scope.go:122] "RemoveContainer" containerID="52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468" Nov 03 01:29:58.786006 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:29:58.785978 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468\": container with ID starting with 52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468 not found: ID does not exist" containerID="52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468" Nov 03 01:29:58.786111 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.786011 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468"} err="failed to get container status \"52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468\": rpc error: code = NotFound desc = could not find container \"52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468\": container with ID starting with 52eecdd70aa44ba2e21ecc816e952bb22162704f1ad734bf0f07b1f389dd1468 not found: ID does not exist" Nov 03 01:29:58.786111 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.786032 4304 scope.go:122] "RemoveContainer" containerID="c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab" Nov 03 01:29:58.788099 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.788074 4304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc93d7-802e-47c6-bcc5-562466d26228-plugins-config-volume\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:58.788099 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.788098 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mslkt\" (UniqueName: \"kubernetes.io/projected/f6cc93d7-802e-47c6-bcc5-562466d26228-kube-api-access-mslkt\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:58.793982 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.793965 4304 scope.go:122] "RemoveContainer" containerID="c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab" Nov 03 01:29:58.794217 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:29:58.794197 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab\": container with ID starting with c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab not found: ID does not exist" containerID="c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab" Nov 03 01:29:58.794284 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.794222 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab"} err="failed to get container status \"c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab\": rpc error: code = NotFound desc = could not find container \"c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab\": container with ID starting with c9eee90de0de5c8d54b8c3a00d5f0090ca01a1de0ef26fd02ef3b651945359ab not found: ID does not exist" Nov 03 01:29:58.936329 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.936302 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv"] Nov 03 01:29:58.939518 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:58.939498 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c9df1-7d7974698kqkdv"] Nov 03 01:29:59.047199 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:59.047132 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c"] Nov 03 01:29:59.050261 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:29:59.050240 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c9df1-854698t65c"] Nov 03 01:30:00.827540 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:00.827496 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b646e7-1966-4903-9141-668dc348de25" path="/var/lib/kubelet/pods/a4b646e7-1966-4903-9141-668dc348de25/volumes" Nov 03 01:30:00.828063 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:00.828027 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6cc93d7-802e-47c6-bcc5-562466d26228" path="/var/lib/kubelet/pods/f6cc93d7-802e-47c6-bcc5-562466d26228/volumes" Nov 03 01:30:10.599751 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:10.599719 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn"] Nov 03 01:30:10.600273 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:10.599944 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" containerID="cri-o://f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50" gracePeriod=30 Nov 03 01:30:15.845653 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.845622 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:30:15.865425 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.865366 4304 generic.go:356] "Generic (PLEG): container finished" podID="1d962085-93d5-4d03-99c9-a3931510c10d" containerID="f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50" exitCode=0 Nov 03 01:30:15.865425 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.865411 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" Nov 03 01:30:15.865656 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.865445 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" event={"ID":"1d962085-93d5-4d03-99c9-a3931510c10d","Type":"ContainerDied","Data":"f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50"} Nov 03 01:30:15.865656 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.865494 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn" event={"ID":"1d962085-93d5-4d03-99c9-a3931510c10d","Type":"ContainerDied","Data":"8c227d86050039b47e194c6e7e07a43126f4cba438cb7c499db6a4ed9ab1d19f"} Nov 03 01:30:15.865656 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.865517 4304 scope.go:122] "RemoveContainer" containerID="f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50" Nov 03 01:30:15.875701 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.875685 4304 scope.go:122] "RemoveContainer" containerID="51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8" Nov 03 01:30:15.883514 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.883498 4304 scope.go:122] "RemoveContainer" containerID="f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50" Nov 03 01:30:15.883846 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:30:15.883787 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50\": container with ID starting with f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50 not found: ID does not exist" containerID="f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50" Nov 03 01:30:15.883846 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.883812 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50"} err="failed to get container status \"f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50\": rpc error: code = NotFound desc = could not find container \"f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50\": container with ID starting with f232565c9764a2f6175dad9c1e715f9de728adffeb86c3354639bad450721e50 not found: ID does not exist" Nov 03 01:30:15.883846 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.883828 4304 scope.go:122] "RemoveContainer" containerID="51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8" Nov 03 01:30:15.884065 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:30:15.884052 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8\": container with ID starting with 51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8 not found: ID does not exist" containerID="51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8" Nov 03 01:30:15.884109 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.884069 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8"} err="failed to get container status \"51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8\": rpc error: code = NotFound desc = could not find container \"51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8\": container with ID starting with 51a5e62f07c884da59987b070ede23c81b83fb8268dde99024654f914d5bade8 not found: ID does not exist" Nov 03 01:30:15.934168 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934148 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-envoy\") pod \"1d962085-93d5-4d03-99c9-a3931510c10d\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " Nov 03 01:30:15.934264 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934201 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-kube-api-access-8ljx7\" (UniqueName: \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-kube-api-access-8ljx7\") pod \"1d962085-93d5-4d03-99c9-a3931510c10d\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " Nov 03 01:30:15.934264 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934230 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-certs\") pod \"1d962085-93d5-4d03-99c9-a3931510c10d\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " Nov 03 01:30:15.934264 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934248 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d962085-93d5-4d03-99c9-a3931510c10d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d962085-93d5-4d03-99c9-a3931510c10d-istio-podinfo\") pod \"1d962085-93d5-4d03-99c9-a3931510c10d\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " Nov 03 01:30:15.934387 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934285 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-istio-token\" (UniqueName: \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-istio-token\") pod \"1d962085-93d5-4d03-99c9-a3931510c10d\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " Nov 03 01:30:15.934387 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934328 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-socket\") pod \"1d962085-93d5-4d03-99c9-a3931510c10d\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " Nov 03 01:30:15.934387 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934380 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-credential-socket\") pod \"1d962085-93d5-4d03-99c9-a3931510c10d\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " Nov 03 01:30:15.934540 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934415 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-data\") pod \"1d962085-93d5-4d03-99c9-a3931510c10d\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " Nov 03 01:30:15.934540 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934455 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d962085-93d5-4d03-99c9-a3931510c10d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d962085-93d5-4d03-99c9-a3931510c10d-istiod-ca-cert\") pod \"1d962085-93d5-4d03-99c9-a3931510c10d\" (UID: \"1d962085-93d5-4d03-99c9-a3931510c10d\") " Nov 03 01:30:15.934540 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934504 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-certs" pod "1d962085-93d5-4d03-99c9-a3931510c10d" (UID: "1d962085-93d5-4d03-99c9-a3931510c10d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:15.934700 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934644 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-socket" pod "1d962085-93d5-4d03-99c9-a3931510c10d" (UID: "1d962085-93d5-4d03-99c9-a3931510c10d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:15.934700 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934664 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-credential-socket" pod "1d962085-93d5-4d03-99c9-a3931510c10d" (UID: "1d962085-93d5-4d03-99c9-a3931510c10d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:15.934825 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934730 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-data" pod "1d962085-93d5-4d03-99c9-a3931510c10d" (UID: "1d962085-93d5-4d03-99c9-a3931510c10d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:15.934903 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934858 4304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-credential-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:30:15.934903 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934878 4304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-data\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:30:15.934903 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934895 4304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:30:15.935063 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934910 4304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-workload-socket\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:30:15.935063 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.934939 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d962085-93d5-4d03-99c9-a3931510c10d-istiod-ca-cert" pod "1d962085-93d5-4d03-99c9-a3931510c10d" (UID: "1d962085-93d5-4d03-99c9-a3931510c10d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:15.936255 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.936233 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-envoy" pod "1d962085-93d5-4d03-99c9-a3931510c10d" (UID: "1d962085-93d5-4d03-99c9-a3931510c10d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:15.936608 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.936584 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d962085-93d5-4d03-99c9-a3931510c10d-istio-podinfo" pod "1d962085-93d5-4d03-99c9-a3931510c10d" (UID: "1d962085-93d5-4d03-99c9-a3931510c10d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:30:15.936675 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.936664 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-kube-api-access-8ljx7" pod "1d962085-93d5-4d03-99c9-a3931510c10d" (UID: "1d962085-93d5-4d03-99c9-a3931510c10d"). InnerVolumeSpecName "kube-api-access-8ljx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:15.936952 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:15.936934 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-istio-token" pod "1d962085-93d5-4d03-99c9-a3931510c10d" (UID: "1d962085-93d5-4d03-99c9-a3931510c10d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:16.036248 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:16.036214 4304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d962085-93d5-4d03-99c9-a3931510c10d-istio-podinfo\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:30:16.036248 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:16.036252 4304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-istio-token\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:30:16.036445 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:16.036267 4304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d962085-93d5-4d03-99c9-a3931510c10d-istiod-ca-cert\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:30:16.036445 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:16.036282 4304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d962085-93d5-4d03-99c9-a3931510c10d-istio-envoy\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:30:16.036445 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:16.036295 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ljx7\" (UniqueName: \"kubernetes.io/projected/1d962085-93d5-4d03-99c9-a3931510c10d-kube-api-access-8ljx7\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:30:16.283379 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:16.283349 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn"] Nov 03 01:30:16.289644 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:16.289621 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c9df1-openshift-default-d64b4f8bf-m4tnn"] Nov 03 01:30:16.827637 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:16.827594 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" path="/var/lib/kubelet/pods/1d962085-93d5-4d03-99c9-a3931510c10d/volumes" Nov 03 01:30:28.655150 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:30:28.655114 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:30:55.120306 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120272 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn"] Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120630 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f6cc93d7-802e-47c6-bcc5-562466d26228" containerName="payload-pre-processing" Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120647 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cc93d7-802e-47c6-bcc5-562466d26228" containerName="payload-pre-processing" Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120661 4304 cpu_manager.go:414] "containerMap: removing container" podUID="a4b646e7-1966-4903-9141-668dc348de25" containerName="payload-processing" Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120666 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b646e7-1966-4903-9141-668dc348de25" containerName="payload-processing" Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120673 4304 cpu_manager.go:414] "containerMap: removing container" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120678 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120688 4304 cpu_manager.go:414] "containerMap: removing container" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120692 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120710 4304 cpu_manager.go:414] "containerMap: removing container" podUID="f184bec5-3fbb-4dbd-b1a8-5b0fff79e532" containerName="maas-api" Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120718 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f184bec5-3fbb-4dbd-b1a8-5b0fff79e532" containerName="maas-api" Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120852 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6cc93d7-802e-47c6-bcc5-562466d26228" containerName="payload-pre-processing" Nov 03 01:30:55.120862 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120869 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b646e7-1966-4903-9141-668dc348de25" containerName="payload-processing" Nov 03 01:30:55.121277 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120880 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" Nov 03 01:30:55.121277 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120890 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d962085-93d5-4d03-99c9-a3931510c10d" containerName="istio-proxy" Nov 03 01:30:55.121277 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.120901 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f184bec5-3fbb-4dbd-b1a8-5b0fff79e532" containerName="maas-api" Nov 03 01:30:55.123557 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.123543 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.224473 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.224447 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-7b3137d3-kserve-self-signed-certs\"" Nov 03 01:30:55.226481 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.226461 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn"] Nov 03 01:30:55.261134 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.261104 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-dshm\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.261280 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.261136 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-model-cache\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.261280 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.261162 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-tmp-dir\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.261365 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.261280 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30ae8e9-7aec-48ea-ab74-71ff98676897-tls-certs\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.261365 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.261333 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-home\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.261430 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.261390 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwfgv\" (UniqueName: \"kubernetes.io/projected/b30ae8e9-7aec-48ea-ab74-71ff98676897-kube-api-access-fwfgv\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.362494 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.362457 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-dshm\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.362494 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.362493 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-model-cache\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.362759 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.362512 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-tmp-dir\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.362759 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.362654 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30ae8e9-7aec-48ea-ab74-71ff98676897-tls-certs\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.362759 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.362706 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-home\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.362966 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.362804 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfgv\" (UniqueName: \"kubernetes.io/projected/b30ae8e9-7aec-48ea-ab74-71ff98676897-kube-api-access-fwfgv\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.362966 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.362897 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-tmp-dir\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.363092 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.362985 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-model-cache\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.363092 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.363033 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-home\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.364980 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.364951 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-dshm\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.365261 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.365240 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30ae8e9-7aec-48ea-ab74-71ff98676897-tls-certs\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.419648 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.419624 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfgv\" (UniqueName: \"kubernetes.io/projected/b30ae8e9-7aec-48ea-ab74-71ff98676897-kube-api-access-fwfgv\") pod \"e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.434564 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.434541 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:55.815505 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:55.815481 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn"] Nov 03 01:30:55.817559 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:30:55.817533 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb30ae8e9_7aec_48ea_ab74_71ff98676897.slice/crio-ffb6102e1dfccaceb9d5de5287783640180e1b81721332bf7df87311e2a1d16c WatchSource:0}: Error finding container ffb6102e1dfccaceb9d5de5287783640180e1b81721332bf7df87311e2a1d16c: Status 404 returned error can't find the container with id ffb6102e1dfccaceb9d5de5287783640180e1b81721332bf7df87311e2a1d16c Nov 03 01:30:56.083420 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:56.083346 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" event={"ID":"b30ae8e9-7aec-48ea-ab74-71ff98676897","Type":"ContainerStarted","Data":"fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27"} Nov 03 01:30:56.083420 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:56.083384 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" event={"ID":"b30ae8e9-7aec-48ea-ab74-71ff98676897","Type":"ContainerStarted","Data":"ffb6102e1dfccaceb9d5de5287783640180e1b81721332bf7df87311e2a1d16c"} Nov 03 01:30:56.196984 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:56.196938 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" podStartSLOduration=1.196923548 podStartE2EDuration="1.196923548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:56.19468034 +0000 UTC m=+5139.478238012" watchObservedRunningTime="2026-11-03 01:30:56.196923548 +0000 UTC m=+5139.480481221" Nov 03 01:30:56.435164 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:30:56.435127 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:30:58.654820 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:30:58.654763 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:31:05.435416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:31:05.435381 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:31:05.447943 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:31:05.447918 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:31:06.183318 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:31:06.183286 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:31:28.734719 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:31:28.734674 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:31:45.471239 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:31:45.471150 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn"] Nov 03 01:31:58.653060 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:31:58.653028 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:32:00.513958 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.513919 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" podUID="b30ae8e9-7aec-48ea-ab74-71ff98676897" containerName="main" containerID="cri-o://fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27" gracePeriod=45 Nov 03 01:32:00.766349 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.766286 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:32:00.809618 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.809597 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-model-cache\") pod \"b30ae8e9-7aec-48ea-ab74-71ff98676897\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " Nov 03 01:32:00.809752 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.809659 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30ae8e9-7aec-48ea-ab74-71ff98676897-kube-api-access-fwfgv\" (UniqueName: \"kubernetes.io/projected/b30ae8e9-7aec-48ea-ab74-71ff98676897-kube-api-access-fwfgv\") pod \"b30ae8e9-7aec-48ea-ab74-71ff98676897\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " Nov 03 01:32:00.809752 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.809685 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-tmp-dir\") pod \"b30ae8e9-7aec-48ea-ab74-71ff98676897\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " Nov 03 01:32:00.809752 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.809708 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-home\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-home\") pod \"b30ae8e9-7aec-48ea-ab74-71ff98676897\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " Nov 03 01:32:00.809752 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.809724 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-dshm\") pod \"b30ae8e9-7aec-48ea-ab74-71ff98676897\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " Nov 03 01:32:00.809980 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.809847 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b30ae8e9-7aec-48ea-ab74-71ff98676897-tls-certs\" (UniqueName: \"kubernetes.io/secret/b30ae8e9-7aec-48ea-ab74-71ff98676897-tls-certs\") pod \"b30ae8e9-7aec-48ea-ab74-71ff98676897\" (UID: \"b30ae8e9-7aec-48ea-ab74-71ff98676897\") " Nov 03 01:32:00.809980 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.809858 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-model-cache" pod "b30ae8e9-7aec-48ea-ab74-71ff98676897" (UID: "b30ae8e9-7aec-48ea-ab74-71ff98676897"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:32:00.810059 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.809992 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-home" pod "b30ae8e9-7aec-48ea-ab74-71ff98676897" (UID: "b30ae8e9-7aec-48ea-ab74-71ff98676897"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:32:00.810059 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.810001 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-tmp-dir" pod "b30ae8e9-7aec-48ea-ab74-71ff98676897" (UID: "b30ae8e9-7aec-48ea-ab74-71ff98676897"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:32:00.810249 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.810229 4304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-tmp-dir\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:32:00.810319 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.810251 4304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-home\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:32:00.810319 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.810259 4304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-model-cache\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:32:00.811983 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.811958 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30ae8e9-7aec-48ea-ab74-71ff98676897-tls-certs" pod "b30ae8e9-7aec-48ea-ab74-71ff98676897" (UID: "b30ae8e9-7aec-48ea-ab74-71ff98676897"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:32:00.811983 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.811978 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-dshm" pod "b30ae8e9-7aec-48ea-ab74-71ff98676897" (UID: "b30ae8e9-7aec-48ea-ab74-71ff98676897"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:32:00.812143 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.812032 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30ae8e9-7aec-48ea-ab74-71ff98676897-kube-api-access-fwfgv" pod "b30ae8e9-7aec-48ea-ab74-71ff98676897" (UID: "b30ae8e9-7aec-48ea-ab74-71ff98676897"). InnerVolumeSpecName "kube-api-access-fwfgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:32:00.911031 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.911004 4304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30ae8e9-7aec-48ea-ab74-71ff98676897-tls-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:32:00.911149 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.911034 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwfgv\" (UniqueName: \"kubernetes.io/projected/b30ae8e9-7aec-48ea-ab74-71ff98676897-kube-api-access-fwfgv\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:32:00.911149 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:00.911048 4304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30ae8e9-7aec-48ea-ab74-71ff98676897-dshm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:32:01.502682 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:01.502654 4304 generic.go:356] "Generic (PLEG): container finished" podID="b30ae8e9-7aec-48ea-ab74-71ff98676897" containerID="fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27" exitCode=0 Nov 03 01:32:01.502889 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:01.502697 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" Nov 03 01:32:01.502889 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:01.502743 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" event={"ID":"b30ae8e9-7aec-48ea-ab74-71ff98676897","Type":"ContainerDied","Data":"fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27"} Nov 03 01:32:01.502889 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:01.502806 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn" event={"ID":"b30ae8e9-7aec-48ea-ab74-71ff98676897","Type":"ContainerDied","Data":"ffb6102e1dfccaceb9d5de5287783640180e1b81721332bf7df87311e2a1d16c"} Nov 03 01:32:01.502889 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:01.502824 4304 scope.go:122] "RemoveContainer" containerID="fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27" Nov 03 01:32:01.511541 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:01.511520 4304 scope.go:122] "RemoveContainer" containerID="fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27" Nov 03 01:32:01.511904 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:32:01.511885 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27\": container with ID starting with fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27 not found: ID does not exist" containerID="fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27" Nov 03 01:32:01.511958 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:01.511913 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27"} err="failed to get container status \"fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27\": rpc error: code = NotFound desc = could not find container \"fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27\": container with ID starting with fb9e952e16ad459013af568d310bf3f9031c633203588e34cbc14741c2106a27 not found: ID does not exist" Nov 03 01:32:01.665453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:01.665422 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn"] Nov 03 01:32:01.713751 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:01.713721 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-7b3137d3-kserve-648b7c5757-btxvn"] Nov 03 01:32:02.828596 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:02.828559 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30ae8e9-7aec-48ea-ab74-71ff98676897" path="/var/lib/kubelet/pods/b30ae8e9-7aec-48ea-ab74-71ff98676897/volumes" Nov 03 01:32:28.654129 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:32:28.654091 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:32:58.653768 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:32:58.653727 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:32:59.035749 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.035660 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-75f546dd9d-jd522"] Nov 03 01:32:59.036130 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.036112 4304 cpu_manager.go:414] "containerMap: removing container" podUID="b30ae8e9-7aec-48ea-ab74-71ff98676897" containerName="main" Nov 03 01:32:59.036195 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.036133 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30ae8e9-7aec-48ea-ab74-71ff98676897" containerName="main" Nov 03 01:32:59.036195 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.036190 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30ae8e9-7aec-48ea-ab74-71ff98676897" containerName="main" Nov 03 01:32:59.038841 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.038821 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" Nov 03 01:32:59.082298 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.082270 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e819a40-5d5e-41a2-9ebb-ad354787b2f9-plugins-config-volume\") pod \"payload-pre-processing-75f546dd9d-jd522\" (UID: \"6e819a40-5d5e-41a2-9ebb-ad354787b2f9\") " pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" Nov 03 01:32:59.082414 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.082335 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l227w\" (UniqueName: \"kubernetes.io/projected/6e819a40-5d5e-41a2-9ebb-ad354787b2f9-kube-api-access-l227w\") pod \"payload-pre-processing-75f546dd9d-jd522\" (UID: \"6e819a40-5d5e-41a2-9ebb-ad354787b2f9\") " pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" Nov 03 01:32:59.092927 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.092903 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 01:32:59.095090 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.095069 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-75f546dd9d-jd522"] Nov 03 01:32:59.142188 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.142163 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-n6gcp\"" Nov 03 01:32:59.183366 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.183343 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l227w\" (UniqueName: \"kubernetes.io/projected/6e819a40-5d5e-41a2-9ebb-ad354787b2f9-kube-api-access-l227w\") pod \"payload-pre-processing-75f546dd9d-jd522\" (UID: \"6e819a40-5d5e-41a2-9ebb-ad354787b2f9\") " pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" Nov 03 01:32:59.183513 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.183413 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e819a40-5d5e-41a2-9ebb-ad354787b2f9-plugins-config-volume\") pod \"payload-pre-processing-75f546dd9d-jd522\" (UID: \"6e819a40-5d5e-41a2-9ebb-ad354787b2f9\") " pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" Nov 03 01:32:59.184091 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.184073 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e819a40-5d5e-41a2-9ebb-ad354787b2f9-plugins-config-volume\") pod \"payload-pre-processing-75f546dd9d-jd522\" (UID: \"6e819a40-5d5e-41a2-9ebb-ad354787b2f9\") " pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" Nov 03 01:32:59.239287 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.239259 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l227w\" (UniqueName: \"kubernetes.io/projected/6e819a40-5d5e-41a2-9ebb-ad354787b2f9-kube-api-access-l227w\") pod \"payload-pre-processing-75f546dd9d-jd522\" (UID: \"6e819a40-5d5e-41a2-9ebb-ad354787b2f9\") " pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" Nov 03 01:32:59.351442 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.351357 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" Nov 03 01:32:59.718452 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.718429 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-75f546dd9d-jd522"] Nov 03 01:32:59.720947 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:32:59.720918 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e819a40_5d5e_41a2_9ebb_ad354787b2f9.slice/crio-aeb2058df120833f3a60ed1570af12c2514e0a6c394f176994fd71bfd3ea4d1b WatchSource:0}: Error finding container aeb2058df120833f3a60ed1570af12c2514e0a6c394f176994fd71bfd3ea4d1b: Status 404 returned error can't find the container with id aeb2058df120833f3a60ed1570af12c2514e0a6c394f176994fd71bfd3ea4d1b Nov 03 01:32:59.828286 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.828251 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" event={"ID":"6e819a40-5d5e-41a2-9ebb-ad354787b2f9","Type":"ContainerStarted","Data":"850c95e9e54b35bb2b3150eec49f6d683135abccf43bc143a52daab43c374251"} Nov 03 01:32:59.828416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.828328 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" event={"ID":"6e819a40-5d5e-41a2-9ebb-ad354787b2f9","Type":"ContainerStarted","Data":"aeb2058df120833f3a60ed1570af12c2514e0a6c394f176994fd71bfd3ea4d1b"} Nov 03 01:32:59.828416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.828372 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" Nov 03 01:32:59.941586 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:32:59.941530 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" podStartSLOduration=1.941514314 podStartE2EDuration="1.941514314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:59.9403713 +0000 UTC m=+5263.223928984" watchObservedRunningTime="2026-11-03 01:32:59.941514314 +0000 UTC m=+5263.225071986" Nov 03 01:33:10.835573 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:33:10.835544 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-75f546dd9d-jd522" Nov 03 01:33:22.311917 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:33:22.311875 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:33:22.312702 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:33:22.312671 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" podUID="5bb4aa28-be0e-4705-9133-23afa41c0701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:33:28.654042 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:33:28.653999 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:33:58.653613 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:33:58.653572 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:34:28.654460 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:34:28.654423 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:34:58.652682 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:34:58.652646 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:35:28.654134 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:35:28.654099 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:35:58.653761 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:35:58.653722 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:36:28.654498 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:36:28.654449 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:36:52.305913 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:36:52.305880 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:36:52.306526 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:36:52.305951 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nk66x" podUID="5bb4aa28-be0e-4705-9133-23afa41c0701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:36:58.655168 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:36:58.655132 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:37:28.653411 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:37:28.653371 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:37:58.654440 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:37:58.654360 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:38:26.281998 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:38:26.281948 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:38:26.282738 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:38:26.282686 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m75z2" podUID="52860790-c4c5-4643-896a-ea677cfc3495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:38:28.654420 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:38:28.654313 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:38:58.669157 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:38:58.669113 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:39:28.654110 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:39:28.654077 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:39:35.640801 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:39:35.639904 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:39:35.640801 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:39:35.639974 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-88r4z" podUID="4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:58.755266 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:39:58.755224 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:40:28.654981 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:40:28.654939 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:40:38.094678 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.094600 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk"] Nov 03 01:40:38.098089 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.098072 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.121724 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.121681 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.121878 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.121800 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbtlq\" (UniqueName: \"kubernetes.io/projected/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-kube-api-access-xbtlq\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.121954 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.121932 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.122004 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.121980 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.202053 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.202026 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b19258-serving-cert\"" Nov 03 01:40:38.204414 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.204392 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk"] Nov 03 01:40:38.222699 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.222675 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.222835 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.222703 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.222835 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.222735 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.222835 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.222810 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbtlq\" (UniqueName: \"kubernetes.io/projected/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-kube-api-access-xbtlq\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.223060 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:40:38.222912 4304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-serving-cert: secret "maas-api-e2e-worker-single-main-b19258-serving-cert" not found Nov 03 01:40:38.223060 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:40:38.222986 4304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-maas-api-tls podName:9b10c5e9-104b-4dc9-b41c-0541cbc81c42 nodeName:}" failed. No retries permitted until 2026-11-03 01:40:38.72296785 +0000 UTC m=+5722.006525520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-maas-api-tls") pod "maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" (UID: "9b10c5e9-104b-4dc9-b41c-0541cbc81c42") : secret "maas-api-e2e-worker-single-main-b19258-serving-cert" not found Nov 03 01:40:38.223462 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.223436 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.225300 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.225279 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.279698 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.279664 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbtlq\" (UniqueName: \"kubernetes.io/projected/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-kube-api-access-xbtlq\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.727077 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.727045 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:38.729571 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:38.729551 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:39.010597 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:39.010508 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:39.385473 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:39.385447 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk"] Nov 03 01:40:39.387639 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:40:39.387609 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b10c5e9_104b_4dc9_b41c_0541cbc81c42.slice/crio-b31a48f7aa478b5bf3fea3b30b1f142251a353bc2d31aa103bd9b7bb58e93421 WatchSource:0}: Error finding container b31a48f7aa478b5bf3fea3b30b1f142251a353bc2d31aa103bd9b7bb58e93421: Status 404 returned error can't find the container with id b31a48f7aa478b5bf3fea3b30b1f142251a353bc2d31aa103bd9b7bb58e93421 Nov 03 01:40:39.389059 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:39.389038 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:40:39.521241 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:39.521207 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" event={"ID":"9b10c5e9-104b-4dc9-b41c-0541cbc81c42","Type":"ContainerStarted","Data":"b31a48f7aa478b5bf3fea3b30b1f142251a353bc2d31aa103bd9b7bb58e93421"} Nov 03 01:40:40.529115 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:40.529083 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" event={"ID":"9b10c5e9-104b-4dc9-b41c-0541cbc81c42","Type":"ContainerStarted","Data":"07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2"} Nov 03 01:40:40.529501 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:40.529149 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:40.642460 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:40.642359 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" podStartSLOduration=1.724228372 podStartE2EDuration="2.642344088s" totalImagesPullingTime="918.115716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:40.640844585 +0000 UTC m=+5723.924402260" watchObservedRunningTime="2026-11-03 01:40:40.642344088 +0000 UTC m=+5723.925901762" Nov 03 01:40:46.540993 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:46.540964 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:40:58.655391 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:40:58.655351 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:40:59.915122 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:59.915087 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r"] Nov 03 01:40:59.918698 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:40:59.918678 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.021495 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.021460 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tls-certs\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.021679 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.021502 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-dshm\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.021679 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.021614 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tmp-dir\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.021679 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.021648 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-model-cache\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.021878 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.021739 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4s84\" (UniqueName: \"kubernetes.io/projected/eb7883d6-1298-4986-aa5f-dd90fb7db71d-kube-api-access-j4s84\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.021878 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.021853 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-home\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.022561 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.022546 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b19258\"/\"openshift-service-ca.crt\"" Nov 03 01:41:00.023038 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.023023 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b19258\"/\"kube-root-ca.crt\"" Nov 03 01:41:00.023373 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.023357 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b19258\"/\"simulator-single-main-b19258-kserve-self-signed-certs\"" Nov 03 01:41:00.023473 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.023415 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b19258\"/\"default-dockercfg-rzkgn\"" Nov 03 01:41:00.025656 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.025631 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r"] Nov 03 01:41:00.122709 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.122677 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4s84\" (UniqueName: \"kubernetes.io/projected/eb7883d6-1298-4986-aa5f-dd90fb7db71d-kube-api-access-j4s84\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.122948 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.122722 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-home\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.122948 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.122747 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tls-certs\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.122948 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.122815 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-dshm\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.122948 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.122915 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tmp-dir\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.122948 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.122942 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-model-cache\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.123193 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.123139 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-home\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.123234 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.123214 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tmp-dir\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.123291 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.123253 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-model-cache\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.125272 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.125249 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-dshm\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.125396 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.125379 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tls-certs\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.178996 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.178933 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4s84\" (UniqueName: \"kubernetes.io/projected/eb7883d6-1298-4986-aa5f-dd90fb7db71d-kube-api-access-j4s84\") pod \"simulator-single-main-b19258-kserve-7544ff5657-sn58r\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.230259 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.230226 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:00.604071 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.604048 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r"] Nov 03 01:41:00.605630 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:41:00.605599 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb7883d6_1298_4986_aa5f_dd90fb7db71d.slice/crio-556ad93199461067121f9d65bccf34847e90c18f752d39b109bbd44e9e3479aa WatchSource:0}: Error finding container 556ad93199461067121f9d65bccf34847e90c18f752d39b109bbd44e9e3479aa: Status 404 returned error can't find the container with id 556ad93199461067121f9d65bccf34847e90c18f752d39b109bbd44e9e3479aa Nov 03 01:41:00.637931 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:00.637896 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" event={"ID":"eb7883d6-1298-4986-aa5f-dd90fb7db71d","Type":"ContainerStarted","Data":"556ad93199461067121f9d65bccf34847e90c18f752d39b109bbd44e9e3479aa"} Nov 03 01:41:01.645370 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:01.645330 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" event={"ID":"eb7883d6-1298-4986-aa5f-dd90fb7db71d","Type":"ContainerStarted","Data":"5b76887ab28535a89e8794b62e5233af2a74d1b0700eea6050be6ab08146b7c8"} Nov 03 01:41:01.758792 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:01.758735 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" podStartSLOduration=2.758723301 podStartE2EDuration="2.758723301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:01.757057752 +0000 UTC m=+5745.040615425" watchObservedRunningTime="2026-11-03 01:41:01.758723301 +0000 UTC m=+5745.042280975" Nov 03 01:41:02.230863 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:02.230821 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:10.230864 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:10.230818 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:10.243839 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:10.243808 4304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:10.701866 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:41:10.701840 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:41:28.656015 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:41:28.655980 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:41:58.656055 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:41:58.656014 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:42:03.773291 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:03.773202 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r"] Nov 03 01:42:05.478358 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:05.478328 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk"] Nov 03 01:42:05.478768 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:05.478527 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" podUID="9b10c5e9-104b-4dc9-b41c-0541cbc81c42" containerName="maas-api" containerID="cri-o://07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2" gracePeriod=30 Nov 03 01:42:06.536097 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.536048 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" podUID="9b10c5e9-104b-4dc9-b41c-0541cbc81c42" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.177:8443/health\": dial tcp 10.133.0.177:8443: connect: connection refused" Nov 03 01:42:06.740634 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.740609 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:42:06.748874 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.748855 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-metrics-certs\") pod \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " Nov 03 01:42:06.748980 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.748888 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-kube-api-access-xbtlq\" (UniqueName: \"kubernetes.io/projected/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-kube-api-access-xbtlq\") pod \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " Nov 03 01:42:06.748980 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.748933 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-maas-api-tls\") pod \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " Nov 03 01:42:06.748980 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.748974 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-openshift-service-ca\") pod \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\" (UID: \"9b10c5e9-104b-4dc9-b41c-0541cbc81c42\") " Nov 03 01:42:06.749378 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.749353 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-openshift-service-ca" pod "9b10c5e9-104b-4dc9-b41c-0541cbc81c42" (UID: "9b10c5e9-104b-4dc9-b41c-0541cbc81c42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:06.751144 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.751114 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-maas-api-tls" pod "9b10c5e9-104b-4dc9-b41c-0541cbc81c42" (UID: "9b10c5e9-104b-4dc9-b41c-0541cbc81c42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:06.751240 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.751224 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-metrics-certs" pod "9b10c5e9-104b-4dc9-b41c-0541cbc81c42" (UID: "9b10c5e9-104b-4dc9-b41c-0541cbc81c42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:06.751287 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.751242 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-kube-api-access-xbtlq" pod "9b10c5e9-104b-4dc9-b41c-0541cbc81c42" (UID: "9b10c5e9-104b-4dc9-b41c-0541cbc81c42"). InnerVolumeSpecName "kube-api-access-xbtlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:06.850070 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.850041 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:06.850182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.850072 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbtlq\" (UniqueName: \"kubernetes.io/projected/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-kube-api-access-xbtlq\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:06.850182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.850088 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:06.850182 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.850101 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10c5e9-104b-4dc9-b41c-0541cbc81c42-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:06.983853 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.983821 4304 generic.go:356] "Generic (PLEG): container finished" podID="9b10c5e9-104b-4dc9-b41c-0541cbc81c42" containerID="07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2" exitCode=0 Nov 03 01:42:06.984029 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.983871 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" Nov 03 01:42:06.984029 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.983897 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" event={"ID":"9b10c5e9-104b-4dc9-b41c-0541cbc81c42","Type":"ContainerDied","Data":"07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2"} Nov 03 01:42:06.984029 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.983953 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk" event={"ID":"9b10c5e9-104b-4dc9-b41c-0541cbc81c42","Type":"ContainerDied","Data":"b31a48f7aa478b5bf3fea3b30b1f142251a353bc2d31aa103bd9b7bb58e93421"} Nov 03 01:42:06.984029 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.983983 4304 scope.go:122] "RemoveContainer" containerID="07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2" Nov 03 01:42:06.993602 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.993585 4304 scope.go:122] "RemoveContainer" containerID="07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2" Nov 03 01:42:06.993860 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:42:06.993840 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2\": container with ID starting with 07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2 not found: ID does not exist" containerID="07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2" Nov 03 01:42:06.993924 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:06.993873 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2"} err="failed to get container status \"07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2\": rpc error: code = NotFound desc = could not find container \"07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2\": container with ID starting with 07834439ee8c2d90ecde5f88054e8eca855b7eaa3b7ab26e6b4ba319af7a74c2 not found: ID does not exist" Nov 03 01:42:07.145270 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:07.145244 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk"] Nov 03 01:42:07.151223 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:07.151200 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b19258-bfb4464dc-kwczk"] Nov 03 01:42:08.828790 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:08.828748 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b10c5e9-104b-4dc9-b41c-0541cbc81c42" path="/var/lib/kubelet/pods/9b10c5e9-104b-4dc9-b41c-0541cbc81c42/volumes" Nov 03 01:42:18.793456 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:18.793396 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" podUID="eb7883d6-1298-4986-aa5f-dd90fb7db71d" containerName="main" containerID="cri-o://5b76887ab28535a89e8794b62e5233af2a74d1b0700eea6050be6ab08146b7c8" gracePeriod=45 Nov 03 01:42:19.044797 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.044701 4304 generic.go:356] "Generic (PLEG): container finished" podID="eb7883d6-1298-4986-aa5f-dd90fb7db71d" containerID="5b76887ab28535a89e8794b62e5233af2a74d1b0700eea6050be6ab08146b7c8" exitCode=0 Nov 03 01:42:19.044917 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.044814 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" event={"ID":"eb7883d6-1298-4986-aa5f-dd90fb7db71d","Type":"ContainerDied","Data":"5b76887ab28535a89e8794b62e5233af2a74d1b0700eea6050be6ab08146b7c8"} Nov 03 01:42:19.044917 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.044863 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" event={"ID":"eb7883d6-1298-4986-aa5f-dd90fb7db71d","Type":"ContainerDied","Data":"556ad93199461067121f9d65bccf34847e90c18f752d39b109bbd44e9e3479aa"} Nov 03 01:42:19.044917 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.044878 4304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="556ad93199461067121f9d65bccf34847e90c18f752d39b109bbd44e9e3479aa" Nov 03 01:42:19.058579 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.058547 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:42:19.184461 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.184433 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-dshm\") pod \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " Nov 03 01:42:19.184627 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.184477 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-model-cache\") pod \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " Nov 03 01:42:19.184682 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.184622 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tls-certs\") pod \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " Nov 03 01:42:19.184682 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.184655 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7883d6-1298-4986-aa5f-dd90fb7db71d-kube-api-access-j4s84\" (UniqueName: \"kubernetes.io/projected/eb7883d6-1298-4986-aa5f-dd90fb7db71d-kube-api-access-j4s84\") pod \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " Nov 03 01:42:19.185453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.184691 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-home\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-home\") pod \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " Nov 03 01:42:19.185453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.184804 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-model-cache" pod "eb7883d6-1298-4986-aa5f-dd90fb7db71d" (UID: "eb7883d6-1298-4986-aa5f-dd90fb7db71d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:19.185453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.184838 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tmp-dir\") pod \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\" (UID: \"eb7883d6-1298-4986-aa5f-dd90fb7db71d\") " Nov 03 01:42:19.185453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.185096 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tmp-dir" pod "eb7883d6-1298-4986-aa5f-dd90fb7db71d" (UID: "eb7883d6-1298-4986-aa5f-dd90fb7db71d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:19.185453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.185152 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-home" pod "eb7883d6-1298-4986-aa5f-dd90fb7db71d" (UID: "eb7883d6-1298-4986-aa5f-dd90fb7db71d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:19.185453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.185261 4304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-home\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:19.185453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.185282 4304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tmp-dir\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:19.185453 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.185296 4304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-model-cache\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:19.187150 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.187118 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-dshm" pod "eb7883d6-1298-4986-aa5f-dd90fb7db71d" (UID: "eb7883d6-1298-4986-aa5f-dd90fb7db71d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:19.187288 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.187270 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tls-certs" pod "eb7883d6-1298-4986-aa5f-dd90fb7db71d" (UID: "eb7883d6-1298-4986-aa5f-dd90fb7db71d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:19.187752 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.187725 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7883d6-1298-4986-aa5f-dd90fb7db71d-kube-api-access-j4s84" pod "eb7883d6-1298-4986-aa5f-dd90fb7db71d" (UID: "eb7883d6-1298-4986-aa5f-dd90fb7db71d"). InnerVolumeSpecName "kube-api-access-j4s84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:19.285817 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.285759 4304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb7883d6-1298-4986-aa5f-dd90fb7db71d-tls-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:19.285817 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.285816 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4s84\" (UniqueName: \"kubernetes.io/projected/eb7883d6-1298-4986-aa5f-dd90fb7db71d-kube-api-access-j4s84\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:19.286034 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:19.285827 4304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb7883d6-1298-4986-aa5f-dd90fb7db71d-dshm\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:20.049885 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:20.049857 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r" Nov 03 01:42:20.088845 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:42:20.088814 4304 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb7883d6_1298_4986_aa5f_dd90fb7db71d.slice/crio-556ad93199461067121f9d65bccf34847e90c18f752d39b109bbd44e9e3479aa\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb7883d6_1298_4986_aa5f_dd90fb7db71d.slice\": RecentStats: unable to find data in memory cache]" Nov 03 01:42:20.210959 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:20.210930 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r"] Nov 03 01:42:20.260122 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:20.260075 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b19258/simulator-single-main-b19258-kserve-7544ff5657-sn58r"] Nov 03 01:42:20.827992 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:42:20.827958 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb7883d6-1298-4986-aa5f-dd90fb7db71d" path="/var/lib/kubelet/pods/eb7883d6-1298-4986-aa5f-dd90fb7db71d/volumes" Nov 03 01:42:28.656138 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:42:28.656101 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:42:58.657152 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:42:58.657118 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:43:28.653686 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:43:28.653646 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:43:58.653725 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:43:58.653641 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:44:28.653841 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:44:28.653804 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:44:58.653518 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:44:58.653481 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:45:09.640410 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.640375 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk"] Nov 03 01:45:09.640842 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.640795 4304 cpu_manager.go:414] "containerMap: removing container" podUID="9b10c5e9-104b-4dc9-b41c-0541cbc81c42" containerName="maas-api" Nov 03 01:45:09.640842 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.640810 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b10c5e9-104b-4dc9-b41c-0541cbc81c42" containerName="maas-api" Nov 03 01:45:09.640842 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.640823 4304 cpu_manager.go:414] "containerMap: removing container" podUID="eb7883d6-1298-4986-aa5f-dd90fb7db71d" containerName="main" Nov 03 01:45:09.640842 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.640829 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7883d6-1298-4986-aa5f-dd90fb7db71d" containerName="main" Nov 03 01:45:09.641003 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.640891 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b10c5e9-104b-4dc9-b41c-0541cbc81c42" containerName="maas-api" Nov 03 01:45:09.641003 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.640899 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7883d6-1298-4986-aa5f-dd90fb7db71d" containerName="main" Nov 03 01:45:09.643941 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.643920 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.719472 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.719441 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/461107a3-1fad-4f42-8aa3-d28d05136d56-openshift-service-ca\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.719614 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.719557 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/461107a3-1fad-4f42-8aa3-d28d05136d56-metrics-certs\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.719614 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.719591 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854x8\" (UniqueName: \"kubernetes.io/projected/461107a3-1fad-4f42-8aa3-d28d05136d56-kube-api-access-854x8\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.719687 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.719649 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/461107a3-1fad-4f42-8aa3-d28d05136d56-maas-api-tls\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.746929 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.746902 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk"] Nov 03 01:45:09.820134 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.820112 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/461107a3-1fad-4f42-8aa3-d28d05136d56-metrics-certs\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.820270 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.820143 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854x8\" (UniqueName: \"kubernetes.io/projected/461107a3-1fad-4f42-8aa3-d28d05136d56-kube-api-access-854x8\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.820270 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.820164 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/461107a3-1fad-4f42-8aa3-d28d05136d56-maas-api-tls\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.820270 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.820203 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/461107a3-1fad-4f42-8aa3-d28d05136d56-openshift-service-ca\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.820901 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.820842 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/461107a3-1fad-4f42-8aa3-d28d05136d56-openshift-service-ca\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.822696 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.822670 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/461107a3-1fad-4f42-8aa3-d28d05136d56-maas-api-tls\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.822820 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.822715 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/461107a3-1fad-4f42-8aa3-d28d05136d56-metrics-certs\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.876012 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.875988 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854x8\" (UniqueName: \"kubernetes.io/projected/461107a3-1fad-4f42-8aa3-d28d05136d56-kube-api-access-854x8\") pod \"maas-api-74648cd7d7-g2grk\" (UID: \"461107a3-1fad-4f42-8aa3-d28d05136d56\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:09.956460 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:09.956438 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:10.326207 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:10.326182 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk"] Nov 03 01:45:10.327833 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:45:10.327807 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461107a3_1fad_4f42_8aa3_d28d05136d56.slice/crio-ac906642ab4096609ca61b8752350ab791fda03e0d14f1942c7c43b780a55dee WatchSource:0}: Error finding container ac906642ab4096609ca61b8752350ab791fda03e0d14f1942c7c43b780a55dee: Status 404 returned error can't find the container with id ac906642ab4096609ca61b8752350ab791fda03e0d14f1942c7c43b780a55dee Nov 03 01:45:11.016742 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:11.016708 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" event={"ID":"461107a3-1fad-4f42-8aa3-d28d05136d56","Type":"ContainerStarted","Data":"ac906642ab4096609ca61b8752350ab791fda03e0d14f1942c7c43b780a55dee"} Nov 03 01:45:12.024959 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:12.024913 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" event={"ID":"461107a3-1fad-4f42-8aa3-d28d05136d56","Type":"ContainerStarted","Data":"61ce19dc29f98d12f699450555384c542449ca662000b4d0ee3d16eba40b1342"} Nov 03 01:45:12.025428 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:12.025125 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:12.138308 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:12.138254 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" podStartSLOduration=2.272216485 podStartE2EDuration="3.138242165s" totalImagesPullingTime="866.02568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:12.136887029 +0000 UTC m=+5995.420444703" watchObservedRunningTime="2026-11-03 01:45:12.138242165 +0000 UTC m=+5995.421799838" Nov 03 01:45:18.034480 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:18.034452 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-g2grk" Nov 03 01:45:18.198999 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:18.198961 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2"] Nov 03 01:45:18.199213 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:18.199173 4304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" podUID="c158dc72-e23a-4f8d-9b7d-53ef9f347281" containerName="maas-api" containerID="cri-o://e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702" gracePeriod=30 Nov 03 01:45:18.401237 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:18.401195 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" podUID="c158dc72-e23a-4f8d-9b7d-53ef9f347281" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.32:8443/health\": dial tcp 10.133.0.32:8443: connect: connection refused" Nov 03 01:45:19.448979 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.448958 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 01:45:19.508609 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.508554 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-metrics-certs\") pod \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " Nov 03 01:45:19.508725 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.508622 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c158dc72-e23a-4f8d-9b7d-53ef9f347281-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c158dc72-e23a-4f8d-9b7d-53ef9f347281-openshift-service-ca\") pod \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " Nov 03 01:45:19.508725 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.508712 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-maas-api-tls\") pod \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " Nov 03 01:45:19.508838 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.508767 4304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c158dc72-e23a-4f8d-9b7d-53ef9f347281-kube-api-access-ps9q4\" (UniqueName: \"kubernetes.io/projected/c158dc72-e23a-4f8d-9b7d-53ef9f347281-kube-api-access-ps9q4\") pod \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\" (UID: \"c158dc72-e23a-4f8d-9b7d-53ef9f347281\") " Nov 03 01:45:19.509045 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.509020 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c158dc72-e23a-4f8d-9b7d-53ef9f347281-openshift-service-ca" pod "c158dc72-e23a-4f8d-9b7d-53ef9f347281" (UID: "c158dc72-e23a-4f8d-9b7d-53ef9f347281"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:19.510671 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.510649 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-metrics-certs" pod "c158dc72-e23a-4f8d-9b7d-53ef9f347281" (UID: "c158dc72-e23a-4f8d-9b7d-53ef9f347281"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:19.510731 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.510696 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-maas-api-tls" pod "c158dc72-e23a-4f8d-9b7d-53ef9f347281" (UID: "c158dc72-e23a-4f8d-9b7d-53ef9f347281"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:19.510879 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.510860 4304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c158dc72-e23a-4f8d-9b7d-53ef9f347281-kube-api-access-ps9q4" pod "c158dc72-e23a-4f8d-9b7d-53ef9f347281" (UID: "c158dc72-e23a-4f8d-9b7d-53ef9f347281"). InnerVolumeSpecName "kube-api-access-ps9q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:19.609884 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.609851 4304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-metrics-certs\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:19.609884 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.609878 4304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c158dc72-e23a-4f8d-9b7d-53ef9f347281-openshift-service-ca\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:19.610048 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.609893 4304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c158dc72-e23a-4f8d-9b7d-53ef9f347281-maas-api-tls\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:19.610048 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:19.609905 4304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps9q4\" (UniqueName: \"kubernetes.io/projected/c158dc72-e23a-4f8d-9b7d-53ef9f347281-kube-api-access-ps9q4\") on node \"ip-10-0-140-172.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:20.070150 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:20.070115 4304 generic.go:356] "Generic (PLEG): container finished" podID="c158dc72-e23a-4f8d-9b7d-53ef9f347281" containerID="e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702" exitCode=0 Nov 03 01:45:20.070336 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:20.070157 4304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" Nov 03 01:45:20.070336 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:20.070199 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" event={"ID":"c158dc72-e23a-4f8d-9b7d-53ef9f347281","Type":"ContainerDied","Data":"e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702"} Nov 03 01:45:20.070336 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:20.070258 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2" event={"ID":"c158dc72-e23a-4f8d-9b7d-53ef9f347281","Type":"ContainerDied","Data":"ae6e395514e02163339dc8950e1bd8a1da51bb64e9370d442155cbbcce6c8f19"} Nov 03 01:45:20.070336 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:20.070292 4304 scope.go:122] "RemoveContainer" containerID="e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702" Nov 03 01:45:20.087928 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:20.087908 4304 scope.go:122] "RemoveContainer" containerID="e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702" Nov 03 01:45:20.088236 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:45:20.088213 4304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702\": container with ID starting with e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702 not found: ID does not exist" containerID="e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702" Nov 03 01:45:20.088330 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:20.088249 4304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702"} err="failed to get container status \"e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702\": rpc error: code = NotFound desc = could not find container \"e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702\": container with ID starting with e6971b2c085f028beb436a5282f22c4f3e9ff13a35499f225f724fbefeea5702 not found: ID does not exist" Nov 03 01:45:20.104998 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:45:20.104974 4304 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc158dc72_e23a_4f8d_9b7d_53ef9f347281.slice\": RecentStats: unable to find data in memory cache]" Nov 03 01:45:20.231837 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:20.231812 4304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2"] Nov 03 01:45:20.281290 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:20.281268 4304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-hdmr2"] Nov 03 01:45:20.828837 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:45:20.828804 4304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c158dc72-e23a-4f8d-9b7d-53ef9f347281" path="/var/lib/kubelet/pods/c158dc72-e23a-4f8d-9b7d-53ef9f347281/volumes" Nov 03 01:45:28.764860 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:45:28.764816 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:45:58.654517 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:45:58.654484 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:46:06.349910 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:46:06.349869 4304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:46:06.350632 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:46:06.350600 4304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2" podUID="ab6138a0-6598-4db3-a290-cbc0d3f67f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:46:28.654350 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:46:28.654315 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:46:58.653918 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:46:58.653838 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:47:18.938960 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:47:18.938919 4304 scope.go:122] "RemoveContainer" containerID="5b76887ab28535a89e8794b62e5233af2a74d1b0700eea6050be6ab08146b7c8" Nov 03 01:47:28.653472 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:47:28.653432 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:47:58.653080 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:47:58.653039 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:48:28.800801 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:48:28.797170 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:48:58.759306 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:48:58.759159 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:49:28.654832 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:49:28.654794 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:49:58.654417 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:49:58.654332 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:50:28.655125 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:50:28.655089 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:50:58.765350 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:50:58.765299 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:51:28.653125 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:51:28.653087 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:51:50.532323 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:51:50.532296 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rttv9_ff259ff5-f5a0-47f7-8a8e-f2f6125748e5/manager/0.log" Nov 03 01:51:51.985057 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:51:51.985029 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dzrml_22dc8845-4b7d-41fb-9c91-cebde8326e19/manager/0.log" Nov 03 01:51:53.252920 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:51:53.252889 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-74648cd7d7-g2grk_461107a3-1fad-4f42-8aa3-d28d05136d56/maas-api/0.log" Nov 03 01:51:58.652875 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:51:58.652839 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:52:00.008320 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:00.008287 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xlr9b_f9b55548-d7c6-4a1f-971b-0b679da43de1/manager/0.log" Nov 03 01:52:00.255684 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:00.255655 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6bknt_1c5959e1-69b6-46e0-94e3-3c6c2e0de70b/manager/0.log" Nov 03 01:52:00.480509 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:00.480485 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-24fmd_71adc051-8d66-4db6-a823-2d912ff1d9ad/kuadrant-console-plugin/0.log" Nov 03 01:52:01.421661 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:01.421633 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-kg9lb_98df725b-22cd-4d9c-b61f-fcaae9c848dc/manager/0.log" Nov 03 01:52:03.269713 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:03.269687 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54744779bc-xvccs_78db11d2-e64f-431e-8460-f4838ff300bf/kube-auth-proxy/0.log" Nov 03 01:52:03.747098 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:03.747054 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-88r4z_4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6/istio-proxy/0.log" Nov 03 01:52:04.614167 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:04.614138 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-m75z2_52860790-c4c5-4643-896a-ea677cfc3495/istio-proxy/0.log" Nov 03 01:52:05.480873 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:05.480846 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nk66x_5bb4aa28-be0e-4705-9133-23afa41c0701/istio-proxy/0.log" Nov 03 01:52:05.732187 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:05.732112 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2_ab6138a0-6598-4db3-a290-cbc0d3f67f05/istio-proxy/0.log" Nov 03 01:52:06.582208 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:06.582153 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-75f546dd9d-jd522_6e819a40-5d5e-41a2-9ebb-ad354787b2f9/payload-pre-processing/0.log" Nov 03 01:52:08.186647 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:08.186622 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-dzqhw_00aa9eb4-4d67-4d73-9660-d7a3469465db/main/0.log" Nov 03 01:52:08.470866 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:08.470738 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-h5npv_7d27c868-cfaa-4629-a287-390559390fc1/main/0.log" Nov 03 01:52:09.186465 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:09.186432 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tb5mm_b71b643c-e306-4416-bd21-293c67b68b1e/main/0.log" Nov 03 01:52:09.895824 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:09.895797 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-vvlq6_49aef8b0-ab85-4ccc-bb3d-c5eebfa27469/main/0.log" Nov 03 01:52:21.211038 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.210995 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26269/must-gather-g7dd9"] Nov 03 01:52:21.211504 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.211485 4304 cpu_manager.go:414] "containerMap: removing container" podUID="c158dc72-e23a-4f8d-9b7d-53ef9f347281" containerName="maas-api" Nov 03 01:52:21.211553 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.211510 4304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c158dc72-e23a-4f8d-9b7d-53ef9f347281" containerName="maas-api" Nov 03 01:52:21.211614 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.211605 4304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c158dc72-e23a-4f8d-9b7d-53ef9f347281" containerName="maas-api" Nov 03 01:52:21.215034 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.215012 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26269/must-gather-g7dd9" Nov 03 01:52:21.271438 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.271414 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26269/must-gather-g7dd9"] Nov 03 01:52:21.278564 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.278533 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d1acc4f-a720-4c49-b176-897bdfe35c52-must-gather-output\") pod \"must-gather-g7dd9\" (UID: \"3d1acc4f-a720-4c49-b176-897bdfe35c52\") " pod="openshift-must-gather-26269/must-gather-g7dd9" Nov 03 01:52:21.278748 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.278726 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s82x9\" (UniqueName: \"kubernetes.io/projected/3d1acc4f-a720-4c49-b176-897bdfe35c52-kube-api-access-s82x9\") pod \"must-gather-g7dd9\" (UID: \"3d1acc4f-a720-4c49-b176-897bdfe35c52\") " pod="openshift-must-gather-26269/must-gather-g7dd9" Nov 03 01:52:21.318580 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.318556 4304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-26269\"/\"default-dockercfg-bj4wk\"" Nov 03 01:52:21.318705 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.318576 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26269\"/\"kube-root-ca.crt\"" Nov 03 01:52:21.318705 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.318581 4304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26269\"/\"openshift-service-ca.crt\"" Nov 03 01:52:21.379331 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.379307 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s82x9\" (UniqueName: \"kubernetes.io/projected/3d1acc4f-a720-4c49-b176-897bdfe35c52-kube-api-access-s82x9\") pod \"must-gather-g7dd9\" (UID: \"3d1acc4f-a720-4c49-b176-897bdfe35c52\") " pod="openshift-must-gather-26269/must-gather-g7dd9" Nov 03 01:52:21.379460 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.379395 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d1acc4f-a720-4c49-b176-897bdfe35c52-must-gather-output\") pod \"must-gather-g7dd9\" (UID: \"3d1acc4f-a720-4c49-b176-897bdfe35c52\") " pod="openshift-must-gather-26269/must-gather-g7dd9" Nov 03 01:52:21.379696 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.379680 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d1acc4f-a720-4c49-b176-897bdfe35c52-must-gather-output\") pod \"must-gather-g7dd9\" (UID: \"3d1acc4f-a720-4c49-b176-897bdfe35c52\") " pod="openshift-must-gather-26269/must-gather-g7dd9" Nov 03 01:52:21.435191 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.435166 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s82x9\" (UniqueName: \"kubernetes.io/projected/3d1acc4f-a720-4c49-b176-897bdfe35c52-kube-api-access-s82x9\") pod \"must-gather-g7dd9\" (UID: \"3d1acc4f-a720-4c49-b176-897bdfe35c52\") " pod="openshift-must-gather-26269/must-gather-g7dd9" Nov 03 01:52:21.530095 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.530038 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26269/must-gather-g7dd9" Nov 03 01:52:21.899448 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.899379 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26269/must-gather-g7dd9"] Nov 03 01:52:21.901715 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:52:21.901690 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d1acc4f_a720_4c49_b176_897bdfe35c52.slice/crio-4d0d43090ba37af6d374888ebaaa0ab40ae12e359ea086893438935cf98d5de4 WatchSource:0}: Error finding container 4d0d43090ba37af6d374888ebaaa0ab40ae12e359ea086893438935cf98d5de4: Status 404 returned error can't find the container with id 4d0d43090ba37af6d374888ebaaa0ab40ae12e359ea086893438935cf98d5de4 Nov 03 01:52:21.903463 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:21.903446 4304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:52:22.774429 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:22.774391 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26269/must-gather-g7dd9" event={"ID":"3d1acc4f-a720-4c49-b176-897bdfe35c52","Type":"ContainerStarted","Data":"4d0d43090ba37af6d374888ebaaa0ab40ae12e359ea086893438935cf98d5de4"} Nov 03 01:52:23.781349 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:23.781309 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26269/must-gather-g7dd9" event={"ID":"3d1acc4f-a720-4c49-b176-897bdfe35c52","Type":"ContainerStarted","Data":"69b4546f3ca0301e20e0425639160c391b28b2b483b56b689f5865328dcf4635"} Nov 03 01:52:23.781795 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:23.781362 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26269/must-gather-g7dd9" event={"ID":"3d1acc4f-a720-4c49-b176-897bdfe35c52","Type":"ContainerStarted","Data":"82d3f015117e458a6d762bc0d384d3361d9af12c10af7d853e06d7edfc70dad0"} Nov 03 01:52:23.894615 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:23.894540 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-26269/must-gather-g7dd9" podStartSLOduration=1.6888820770000001 podStartE2EDuration="2.894523168s" totalImagesPullingTime="1.205641091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:23.894254509 +0000 UTC m=+6427.177812181" watchObservedRunningTime="2026-11-03 01:52:23.894523168 +0000 UTC m=+6427.178080834" Nov 03 01:52:28.656611 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:52:28.656571 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:52:31.079858 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:31.077338 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dz9d5_ea5a64df-b2ce-4bd0-b4eb-97144bcb9aad/global-pull-secret-syncer/0.log" Nov 03 01:52:33.181443 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:33.181387 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rttv9_ff259ff5-f5a0-47f7-8a8e-f2f6125748e5/manager/0.log" Nov 03 01:52:33.571853 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:33.571756 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rlgnn_434003bf-315d-42b3-8ee4-9cdc53936d73/konnectivity-agent/0.log" Nov 03 01:52:34.823098 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:34.823039 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-172.us-west-1.compute.internal_3a634ffde00653c750ef568ec11d5617/haproxy/0.log" Nov 03 01:52:35.528964 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:35.528917 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dzrml_22dc8845-4b7d-41fb-9c91-cebde8326e19/manager/0.log" Nov 03 01:52:39.015881 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:39.015835 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rttv9_ff259ff5-f5a0-47f7-8a8e-f2f6125748e5/manager/0.log" Nov 03 01:52:39.802628 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:39.800403 4304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26269/perf-node-gather-daemonset-k52hf"] Nov 03 01:52:39.811759 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:39.811728 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:39.893944 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:39.893902 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26269/perf-node-gather-daemonset-k52hf"] Nov 03 01:52:39.950105 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:39.950071 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g5sv\" (UniqueName: \"kubernetes.io/projected/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-kube-api-access-8g5sv\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:39.950297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:39.950175 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-proc\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:39.950297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:39.950199 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-podres\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:39.950416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:39.950313 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-lib-modules\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:39.950416 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:39.950339 4304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-sys\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.051186 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.051155 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-lib-modules\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.051186 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.051185 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-sys\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.051696 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.051219 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g5sv\" (UniqueName: \"kubernetes.io/projected/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-kube-api-access-8g5sv\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.051696 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.051295 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-sys\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.051696 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.051321 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-proc\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.051696 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.051348 4304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-podres\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.051696 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.051297 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-lib-modules\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.051696 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.051409 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-proc\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.051696 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.051440 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-podres\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.107492 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.107435 4304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g5sv\" (UniqueName: \"kubernetes.io/projected/eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc-kube-api-access-8g5sv\") pod \"perf-node-gather-daemonset-k52hf\" (UID: \"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.126295 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.126272 4304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:40.508796 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.506336 4304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26269/perf-node-gather-daemonset-k52hf"] Nov 03 01:52:40.512756 ip-10-0-140-172 kubenswrapper[4304]: W1103 01:52:40.512726 4304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeeff32d2_59bb_4ac5_8b1e_2d7e5be48dfc.slice/crio-54688c5afda9e89d68c6555f69b6aca606a6b3f0bf7c3dbc8d4c2121e495b5cd WatchSource:0}: Error finding container 54688c5afda9e89d68c6555f69b6aca606a6b3f0bf7c3dbc8d4c2121e495b5cd: Status 404 returned error can't find the container with id 54688c5afda9e89d68c6555f69b6aca606a6b3f0bf7c3dbc8d4c2121e495b5cd Nov 03 01:52:40.683307 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.683275 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xlr9b_f9b55548-d7c6-4a1f-971b-0b679da43de1/manager/0.log" Nov 03 01:52:40.948224 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.948189 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" event={"ID":"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc","Type":"ContainerStarted","Data":"a3de4b1d604b5cdf2d92c0ad59b42ded43bb4d80d9e0415cb941c4865167da1e"} Nov 03 01:52:40.948427 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.948236 4304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" event={"ID":"eeff32d2-59bb-4ac5-8b1e-2d7e5be48dfc","Type":"ContainerStarted","Data":"54688c5afda9e89d68c6555f69b6aca606a6b3f0bf7c3dbc8d4c2121e495b5cd"} Nov 03 01:52:40.948427 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:40.948281 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:41.060934 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:41.060888 4304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" podStartSLOduration=2.060872736 podStartE2EDuration="2.060872736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:41.06036084 +0000 UTC m=+6444.343918519" watchObservedRunningTime="2026-11-03 01:52:41.060872736 +0000 UTC m=+6444.344430410" Nov 03 01:52:41.097276 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:41.097245 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6bknt_1c5959e1-69b6-46e0-94e3-3c6c2e0de70b/manager/0.log" Nov 03 01:52:41.362258 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:41.362181 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dzrml_22dc8845-4b7d-41fb-9c91-cebde8326e19/manager/0.log" Nov 03 01:52:41.512570 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:41.512539 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-24fmd_71adc051-8d66-4db6-a823-2d912ff1d9ad/kuadrant-console-plugin/0.log" Nov 03 01:52:42.840889 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:42.840859 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-kg9lb_98df725b-22cd-4d9c-b61f-fcaae9c848dc/manager/0.log" Nov 03 01:52:44.818372 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:44.818338 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rttv9_ff259ff5-f5a0-47f7-8a8e-f2f6125748e5/manager/0.log" Nov 03 01:52:46.965811 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:46.965757 4304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-26269/perf-node-gather-daemonset-k52hf" Nov 03 01:52:47.146996 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:47.146966 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dzrml_22dc8845-4b7d-41fb-9c91-cebde8326e19/manager/0.log" Nov 03 01:52:53.277697 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:53.277664 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7s2c_ea3a3688-008d-4735-b23a-ef2bb50254b1/node-exporter/0.log" Nov 03 01:52:53.685297 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:53.685267 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7s2c_ea3a3688-008d-4735-b23a-ef2bb50254b1/kube-rbac-proxy/0.log" Nov 03 01:52:53.784260 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:53.784226 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54744779bc-xvccs_78db11d2-e64f-431e-8460-f4838ff300bf/kube-auth-proxy/0.log" Nov 03 01:52:54.095885 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:54.095807 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7s2c_ea3a3688-008d-4735-b23a-ef2bb50254b1/init-textfile/0.log" Nov 03 01:52:54.213488 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:54.213458 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-88r4z_4ecd91cb-b063-43aa-9a47-b7dc57a6a2c6/istio-proxy/0.log" Nov 03 01:52:54.957095 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:54.957070 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-m75z2_52860790-c4c5-4643-896a-ea677cfc3495/istio-proxy/0.log" Nov 03 01:52:55.598436 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:55.598402 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nk66x_5bb4aa28-be0e-4705-9133-23afa41c0701/istio-proxy/0.log" Nov 03 01:52:55.778283 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:55.778227 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2d4dbde-f2df-4182-994e-aa841a180fa9/prometheus/0.log" Nov 03 01:52:55.809947 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:55.809919 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2_ab6138a0-6598-4db3-a290-cbc0d3f67f05/istio-proxy/0.log" Nov 03 01:52:56.189032 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:56.189001 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2d4dbde-f2df-4182-994e-aa841a180fa9/config-reloader/0.log" Nov 03 01:52:56.447626 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:56.447558 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-75f546dd9d-jd522_6e819a40-5d5e-41a2-9ebb-ad354787b2f9/payload-pre-processing/0.log" Nov 03 01:52:56.608409 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:56.607509 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2d4dbde-f2df-4182-994e-aa841a180fa9/thanos-sidecar/0.log" Nov 03 01:52:57.013050 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:57.013019 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2d4dbde-f2df-4182-994e-aa841a180fa9/kube-rbac-proxy-web/0.log" Nov 03 01:52:57.424890 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:57.424851 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2d4dbde-f2df-4182-994e-aa841a180fa9/kube-rbac-proxy/0.log" Nov 03 01:52:57.838584 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:57.838488 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2d4dbde-f2df-4182-994e-aa841a180fa9/kube-rbac-proxy-thanos/0.log" Nov 03 01:52:58.251000 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:58.250937 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f2d4dbde-f2df-4182-994e-aa841a180fa9/init-config-reloader/0.log" Nov 03 01:52:58.657088 ip-10-0-140-172 kubenswrapper[4304]: E1103 01:52:58.657048 4304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-898tb" containerName="extractor" Nov 03 01:52:59.497021 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:52:59.496985 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-w7r22_2c71cdcf-0223-41f4-8a03-a3776968ca02/prometheus-operator-admission-webhook/0.log" Nov 03 01:53:02.076406 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:53:02.076379 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xlr9b_f9b55548-d7c6-4a1f-971b-0b679da43de1/manager/0.log" Nov 03 01:53:02.288488 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:53:02.288459 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6bknt_1c5959e1-69b6-46e0-94e3-3c6c2e0de70b/manager/0.log" Nov 03 01:53:02.499581 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:53:02.499555 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-24fmd_71adc051-8d66-4db6-a823-2d912ff1d9ad/kuadrant-console-plugin/0.log" Nov 03 01:53:03.295339 ip-10-0-140-172 kubenswrapper[4304]: I1103 01:53:03.295304 4304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-kg9lb_98df725b-22cd-4d9c-b61f-fcaae9c848dc/manager/0.log"